ProxyFusion: Face Feature Aggregation Through Sparse Experts
Authors: Bhavin Jawade
Alexander Stone
Deen Dayal Mohan
Xiao Wang
Srirangaraj Setlur
Venu Govindaraju
Venue: NeurIPS 2024 poster (2024)
Keywords: Feature Fusion
Face Recognition
Pooling
Primary area: deep_learning_architectures
TL;DR: Efficient face feature fusion using sparse experts for robust recognition in challenging environments.

Abstract: Face feature fusion is indispensable for robust face recognition, particularly in scenarios involving long-range, low-resolution media (unconstrained environments) where not all frames or features are equally informative. Existing methods often rely on large intermediate feature maps or face metadata information, making them incompatible with legacy biometric template databases that store pre-computed features. Additionally, real-time inference and generalization to large probe sets remains challenging. 
To address these limitations, we introduce a linear time O(N) proxy based sparse expert selection and pooling approach for context driven feature-set attention. Our approach is order invariant on the feature-set, generalizes to large sets, is compatible with legacy template stores, and utilizes significantly less parameters making it suitable real-time inference and edge use-cases. Through qualitative experiments, we demonstrate that ProxyFusion learns discriminative information for importance weighting of face features without relying on intermediate features. Quantitative evaluations on challenging low-resolution face verification datasets such as IARPA BTS3.1 and DroneSURF show the superiority of ProxyFusion in unconstrained long-range face recognition setting. 
Our code and pretrained models are available at: https://github.com/bhavinjawade/ProxyFusion

ProxyFusion: Face Feature Aggregation Through
Sparse Experts
Bhavin Jawade University at Buffalo
Alexander Stone University at Buffalo austone@buffalo.cm
Deen Dayal Mohan University at Buffalo dmohan@buffalo edu
bhavinjabbuffall edu
Xiao Wang University Buffalo xwang2778buffalo. edu
Srirangaraj Setlur University at Buffalo setlurObuffalo. edu
Venu Govindaraju University Buffalo govind@buffalo edu
Abstract
Face feature fusion indispensable for robust face recognition, particularly in scenarios involving long-range, low-resolution media (unconstrained environments) where not all frames or features are equally informative. Existing methods often rely on large intermediate feature maps face metadata information, making them incompatible with legacy biometric template databases that store pre-computed features. Additionally, real-time inference and generalization to large probe sets remains challenging. To address these limitations, we introduce linear time O (N) proxy based sparse expert selection and pooling approach for context driven feature-set attention. Our approach order invariant on the feature-set, generalizes to large sets, is compatible with legacy template stores, and utilizes significantly less parameters making it suitable real-time inference and edge use- cases. Through qualitative experiments, demonstrate that ProxyFusion learns discriminative information for importance weighting of face features without relying on intermediate features. Quantitative evaluations on challenging low- resolution face verification datasets such as IARPA BTS3.1 1 and DroneSURF show the superiority of ProxyFusion in unconstrained long-range face recognition setting. Our code and pretrained models are available at: https: /ttps:://gthubb /gtgttttt.......//// com/ bhavinj awade awadd//rooxFusioo
Introduction
Face recognition (FR) involves generating representations templates from face images for 1:1 verification and I:N identification between query media, known as the probe, and an enrolled biometric template, known the gallery. Numerous studies have explored novel feature extraction architectures [10] and metric dearning-based loss formulations [7] for learning discriminative representations. But, feature extraction is typically the first step in building face matching system; the second step involves creating robust templates using sets of images or videos of the same individual. This requires fusing aggregating representations from different face images (or frames, in the case of videos) to obtain unified template.
Recently, there has been heightened interest in recognizing individuals under extremely challenging conditions, such as from long distances and high altitudes, exemplified by the IARPA BRIAR program. These scenarios introduce novel challenges, making face feature aggregation only a limited set of frames would contain discriminative identity information. In cross-distribution setting, such low-resolution high-resolution face matching where significant distribution gap
crucial since
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Liner Campleddy
Cumpusbbee with Legay
Chap Donatn
Complesity with Large Probe Set
Proc/Padce
V
NAIN]
Poll
Tempate
Istrsedd Attentinn (os CaNAA, BSA)
X
Scull Prdde det
Prote Set
Others
Gullery
Rosed Metaads (ng. CAthon PFT!
X
X
V
Maradars Approachee [ca: MPAN)
X
X
X
PronyPaaties (Duns)
Figure I: We design our approach to solve three primary challenges (i) Cross-Domain Matching: Matching low-resolution, long-range faces with high-quality gallery faces. (ii) Linear Runtime Complexity: Ensuring our method's time complexity increases linearly with the number of features. (iii) Compatibility with Legacy Templates: Relying solely on final feature vectors for fusion to maintain compatibility with pre-enrolled feature stores that lack intermediate features or metadata.
exists, face feature fusion becomes essential for selecting the most relevant gallery or probe frames for accurate matching.
Existing face feature fusion methods face several key challenges: (i) generalization to large probe set [9) (ii) real-time inference with low computational cost. (iii) compatibility with legacy template stores, and (iv) cross-distribution matching capabilities [5] Additionally, these methods should perform importance attribution based on feature quality and feature frequency. This work aims to train a fusion model that learns an order and length-invarian feature weighting strategy. Specifically, given an unordered set of N features, the model should return a set scores that, when used to weigh the original features, produce aggregated vector that robustly represents the feature set while maximizing identity-specific information.
As observed by CAFace [9]. typical set-to-set attention mechanisms such as Multihead Attention (MHA) and other intra-set attention methods like RSA [2] or CoNAN [S] exhibit quadratic time complexity. ((N2). This makes them unsuitable for feature aggregation when dealing with large probe sets. Furthermore, methods such as CAFace [9] require high-dimensional intermediate feature maps (H x W x C) from the face feature extractor to compute the style information for feature importance weighting. Other approaches, such as [44 [3, leverage external metadata predictors to extract facial characteristics like pose, gender, and distance, which are then used for attribute- conditioned aggregation. These methods are incompatible with existing biometric template stores which typically only store the penultimate features.
In this work we propose novel feature aggregation framework, ProxyFusion (PF) that utilizes K learnable proxies P, (where P, E R xD) to implicitly represent latent facial attributes. These learnable proxies are used for selecting pooling experts based on relevancy scores, Inspired by works on offture-of-expprts 4 and its sparse variants [16], we utilize only the most relevant K experts for each feature-set, thereby reducing the inference time parameters. The proposed pooling experts generate set-centers conditioned on the feature-set distribution. Divergence of input set features from set-centers represents their informativeness. We utilize the divergence to pool the N feature vectors into one aggregated representation.
As we will discuss later, ProxyFusion approach performs order-invariant, size-agnostic feature aggregation without relying on digeedimensional intermediate feature maps additional metadata. Through various qualitative experiments, we demonstrate that experts can learn distinctive face quality information. Since the effectiveness of face feature fusion is particularly evident long-range, low-resolution settings, we conduct extensive experiments 0. which includes videos and images collected in extreme unconstrained environments, along with their constrained counterparts. In addition to BTS3. I, we also conducted experiments on another unconstrained UAV distrrbutiiiinonn dataset, DroneSURF (6).
the IARPA's BRIAR BTS3.1 1 dataset
In summary, the key contributions of this paper are:
We introduce ProxyFusion, a feature aggregation framework using learnable query em- beddings to select pooling experts. The methods avoids the need for high-dimensional

---

Feature Extrction
Expert Network Selectinn
Sparse Expert Network Fealurt Appregation
OOPCO
R
Figure 2: An overview of our proposed ProxyFusion Approach. Post feature extraction, our method divided two end-to-end trainable stages: (i) Expert Selection and (ii) Sparse Expert Network Feature Aggregation. The Expert Selection module takes the {f and returns the indices expert networks based on proxy relevancy scores. Next, the selected expert networks compute set-centers conditioned on distribution and aligned proxy. These set-centers attend over the input feature set to compute aggregation weights.
{f I
intermediate feature maps additional metadata making it compatible with existing bio- metric template stores.
By selecting the top experts for each feature set inspired by the mixture-of-experts approach, ProxyFusion significantly reduces inference time parameters.
Extensive qualitative and quantitative experiments challenging datasets (IARPA BRIAR BTS3.1 I and DroneSURF) show ProxyFusion's 's effectiveness in long-range, low-resolution face matching, improving FR performance in extreme environments.
Method
The objective of our method correctly match the subject identities in probe media the subject identities in gallery media. Our problem setting deviates from more conventional face recognition because have additional challenges where the probe images have significantly degraded quality as they exist in the long-range, low-resolution domain, where as the gallery images are in the close-range, high-resolution domain. Additionally, the probe feature for given subject can be extremely large, possibly in the hundreds of thousands. We define these two sets of images as Ig = {G1, G2 b2...... G. for the set of high quality gallery images, and Ip {P1 P2...., Pa) for the set of low quality probe images. Since face feature set is composed of faces in different poses and quality, they have different degrees of discriminative identity information. The goal of
aggregating features across the feature set [ is to form a template vector t E R R-dd that best represents the subject's identity for cross- coos--isstribution matching.
Figure illustrates our proposed approach. Since our method strictly focuses on feature fusion of output embeddings, we pre-extrace face features using frozen pretrained face detection and recognition backbones. The face detector is denoted as fD I D, where I is a set of frames and D is the set of detected face regions. For our face feature extractor, denote the model as fr. D E, which maps a cropped face region in to a d-dimensional feature embedding in the set & = {f. N where is face feature vector.
Our proposed feature fusion method composed of two key parts. The first involves sparsely selecting relevant experts, and the second performs feature pooling using the chosen experts. In section 2.1 we discuss the Expert Network Selection strategy followed by the feature aggregation apporach using these experts.
2.1 Expert Network Selection
Given a set & of face features (f) we define set of learnable proxies {p,} where f; RRd. Here proxies are fixed dimensional embeddings that would represent latent information about facial characteristics required to decide which expert network should be utilized. We project each feature f, and proxy P, to unified latent space using shared projection layer, represented by the
3

---

matrix W Rdxd";
f Rd, Pj= Rd
where d' the output dimensionality the W. We choose d' << d for parameter efficiency.
Using f/ Pjz we compute the Proxy Relevancy Scores denoted by ry E IR, by computing the similarities between each projected proxy P/ and all the projected features fy, and accumulating these similarities across all features rj > (D': .f.))
To perform expert network selection, index the top-K values from the set of proxy relevancy scores {r,}ý= We denote these indices as Teo-* = -2. J.), and use them to selectively
activate a subset of expert networks {
I S {E,) where K < K. These selected expert networks will be used in the following stage for feature pooling.
2.2 Sparse Expert Network Feature Aggregation
We subsampled proxies and their associated expert networks using their relevancy scores with respect to the feature-set in the previous stage. Here we describe our approach to use these sparsely selected expert networks to extract conditional set-centers for aggregating features into the final template vector representation. Motivated by mixture-of-expert homogeneous experts that rank features differently based on learned implicit characteristics. We condition our expert networks over the learned subsampled proxies along with the cross cross-sample mean and variance across all dimensions within the feature set. More concretely, given set of feature vectors {fi where each vector f, E Rd, we compute the mean and variance vectors E Rd o² E Rd as follows:
[4), we aim to learn mutually exclusive yet
E f o² N Σ f 11)2
then, we create our input feature distribution representation as x, = [M θ o² + P.). where x; E R3-d and ① denotes concatenation. For each xj. we infer through its corresponding expert network, determined by the associated proxy, to obtain the set-centers {c, {c,)K= where c (xy), Vj {I....,R).
The outputs of the expert networks, referred here set-centers, are used to compute the divergence of each feature in the original set. These divergence scores are then utilized to compute feature importance. We compute the divergence scores the un-normalized alignment between the features [ and set-centers {c, given by Cj f.. Next, we softmax these divergence scores to compute weights as follows:
exp(c, f.)
aij
Wi (I,...,NN..i (I....,RR)
Σ N exp(c, fx)
Finally, we compute the weighted sum of the feature vectors for each set-center using the weights agz:
Sy aigh, Vj (1.
where {s,} is the set of reduced aggregated representation from the selected experts. We define
the final template vector t as concatenation of s, given by: t t [1 || S2.......... sR]--
Set Length and Order Invariance: Feature set length invariance refers to our method's ability to effectively handle feature sets of any size, while feature order invariance indicates that our method can process any permutation of features without affecting the outcome. Feature set length invariance arises through the application of mean and variance as representation of feature-set distribution as also utilized by [5]. Mean and variance could generalize to varying feature lengths the training-set consists of diversely sampled features. We facilitate this through our batch creation strategy.
To construct batch, we select set of M subject identities LM { and subsample their respective probe and gallery feature sets. Let P; denote the probe feature set for identity I; and G, denote the
respective gallery feature set. We define the subsets P. P, and G. G., where the sizes of these

---

subsets are uniformly sampled from the range IPil. G ~ ULL, U). For efficient computation, we zero-pad the maximum feature set length for any identity within batch create uniformly sized tensors.
The order independence of features is direct consequence of the properties of mean and variance. This invariance also applies to our Proxy Relevancy Scores, {r, {ry}カー because the order in which the feature set is multiplied with the proxies is not relevant. The sum computed along the dimension of the feature vectors, identifying the most relevant proxy without being affected by feature order.
2.3 Optimization
We primarily optimize aggregation experts and the proxies with the identification objective. The identification loss addresses the primary goal to bridge the distribution gap between the probe and gallery templates for same identity while increasing the inter-class variance across different identities.
The output of ProxyFusion is the template t. For all feature subsets in a batch we compute the identity loss using the supervised contrastive loss 77. Let, (tP,, ,t be the batch consisting of probe and gallery features then Lia is defined as:
exp(t: (2/rr) ESEA( exp(t: 1//T)
Lu 4u= = Σ P()) IEB
Σ PEP())
where t, the feature embedding index in the the (t", t²). P(i) is the set of indices of samples with the same subject label as i and A(i) is the set of indices of all samples with identities different from subject
Here, T is the softmax's temperature parameter.
During training we want to supervise the experts to learn focus on mutually exclusive information through the proxy conditioning. Additionally, while computing the proxy relevancy scores, want different proxies to attend over different facial characteristics. To achieve this, we propose an additional optimization criteria referred here the proxy loss. We will fix K uniformly spaced equidistant vectors on the unit hypersphere in R We choose K dimensions because it is the lowest dimensional space such that our proxies can be equidistant yet farthest apart from each K other. We project the proxies p E Rd down using Wp e RK- Then, we fix K vectors {v:} to be equidistant from each other as follows:
V. (e. (e. - (c
vd (L....,K-- -- -I,VE {L....,K}
where e; E RR are the standard basis vectors. Based on this the proxy loss is defined as:
K
Lproxy
1
(1 + exp x))) +
Σ In (1 + exp (B(sa - X)))
K
K
KEK
Where, 8: = (p; v&) (Ipp:ll (vel) is the similarity between the 7hh proxy and hthh fixed basis vector. is the threshold for the exponential function. The first part of this term enforces that proxies get closer to their respective basis vectors, while second part enforces that they move away from all other negative basis vectors.
The final loss given by = LID + 7 Crroxy, where 7 is weightage of the proxy loss.
Experiments
3.1 Dataset
To illustrate the effectiveness of our aggregation technique long-range, low-resolution, uncon- strained matching scenarios, selected challenging datasets featuring low-quality images and
5

---

videos captured from long distances. Our experiments utilize the following datasets for training: (i) BRIAR Research Set 3 (BRS 3) This dataset is from IARPA's BRIAR program Phase 1, featuring videos and images from 170 participants in controlled and field settings. Controlled settings have high-resolution facial images at close range, while field settings include media captured from 100 to 500 meters away. For training CoNAN, we used 49,429 video clips and images from BRS 3, with 780 field-setting clips, 23, 23,489 controlled-sctting clips, and 5, images. Our method is trained on BRS 3 for fair comparison with methods like [5] trained on the same dataset. (ii) WebFace 4M [22]: Apart from BRIAR, we also present results by training our method WebFace 4M dataset. This is done present fair evaluation with CAFace [9] which has been trained on WebFace 4M. Following CAFace 19], we use their randomly sampled subset, consisting of 813, 482 images from 10, 000 identities to train our aggregation function. Following [5), we evaluate our method on following two datasets: (i) BTS 3.1: This is the test set for IARPA BRIAR Phase 1 evaluation. We report results for the face-includer treatment and control protocols. The treatment set has 5,822 probe videos from 260 subjects in uncontrolled settings, while the control set has 1,914 probe videos from 256 subjects in regulated settings. The BTS 3 gallery is split into Gallery 1 (47,925 clips/images, 485 subjects) and Gallery 2 (47,4 (47,413 clips/images, 481 subjects), with 351 common distractor identities. (ii) DroneSURF: This dataset includes Active and Passive Surveillance settings, each with 100 videos and 786,000+ face annotations. Following [12], we split subjects randomly: 60% (34 identities) for trining/validationn 40% (24 identities) for testing. The dataset has 200 videos of 58 subjects, over 41 1,000 drone- cope-ca frames. Results are based on the video-wise identification protocol.
3.2
Implementation Details
Face Detector and Alignment:
We present results using two popular face detectors: MTCNN 21 and RetinaFace (2 Unless otherwise specified, the results reported using the RetinaFace.
Feature Extractors: Following previous works [5) [ we report results using two pretrained frozen feature extractors- Adaface 3 and Areface 3. For Adaface [8] we use ResNet- model pre- trained on WebFace. Each face image is resized to 1 112x1 12x3 before feature extraction. For Arcface, we extract features using a MSV2 pre-trained ResNet-50 backbone. For fusion architecture trained we utilize the precomputed features provided by previous
using WebFace 4M Adaface features [8,
workd (9] for fair comparison.
Architecture and Hyperparameters: For the expert networks we utilize a three layer MLP with LeakyReLU activation and dropout with probability of 0.5. More specifically, the first layer in the MLP projects from 1536 to 1024, the second layer from 1024 to 1024, and last layer from 1024 to 512. Overall the MLP has 3.14M learnable parameters. For supervised contrastive loss, we use a temperature of 0.1. The proxy loss weight = 0.01 and threshold À = 0.1. We utilize the Adafactor with adaptive learning rates as the optimizer. For all SoTA experiments we utilize number of total proxies K as 11, and number of selected experts K as 4. We choose the number of identities in a batch M = 170 based on available GPU memory. We choose the bounds for probe and gallery subset sizes to be L = 100 and U = 1200. All experiments are performed on 1 x A6000 48GB NVIDIA GPU. Most experiments require nearly 2 hours with precomputed features.
3.3 Discussion and Ablation
Weight Visualizations and Interpretability: In Fig. 3] we illustrate the weightings assigned by the set-centers extracted from each selected expert for the face embeddings. This visualization offers insights into the aggregation function's 's capacity to discern the informativeness of faces. Notably, the model operates solely on frozen precomputed features within the feature set, with no access to intermediate features or metadata.
As evidenced in Fig. 3 each expert attributes higher weights to faces that exhibit more discriminative identity information, such as frontal or profile views, in both the gallery and probe sets. Conversely, the model assigns minimal weight to poor-quality crops, such as images of the back of the head or inaccurate detections. In the case of probes, which are collected in unconstrained, long-range settings, only limited number of frames in video contain significant facial information. Hence, the model assigns considerably higher weights to these informative frames within the probe set (e.g., Expert 10 assigns approximately 0.007205 to a frontal face).
In contrast, the gallery set comprises numerous high-quality, informative faces, resulting in the model assigning relatively lower weights to these frames due to their abundance. Consequently, weight

---

Expert
Expert
Gallery
Expert
Expert
Probes
Expert
Expert
Figure 3: Visualizations learned weights on BTS3.1 dataset's gallery and probe set. Images on the top are from high quality gallery, and images on the bottom are from low resolution long-range probes. Faces are sorted based ProxyFusion attention weights from low high. We present these weights for each of the selected expert.
Number of Experts
65
033
03.4
621
62
622
603
63.
612
612
10
14
Number of Total Proxics Total Experts
Figure 4: A heatmap of TAR@FAR=1002 on Face Included Treatment Setting of BTS 3.1. The X-axis is the number selected experts while the Axis total number of experts proxies.
assignments decrease more gradually in the gallery set compared to the probe set. Additionally, each expert provides subtly different weightings to the same frames, enhancing the overall representation of the face. This analysis demonstrates that the ProxyFusion model adeptly discerns the informativeness of facial features in interpretable yet effective manner.
Effect of Number of Proxies: In Figure 4 we present the face verification performance in terms of TAR @ FAAR=10 plotted against the number of proxies K (X-axis) and the selected number of experts K (Y-axis). The results indicate that increasing the number of proxies generally enhances model performance. However, this improvement plateaus around 10-12 proxies, and further increasing the number of proxies beyond 13 leads to overfitting. This overfitting likely occurs because the model, with higher number of experts during training, becomes more prone to fitting the training distribution and training subjects too closely, thus failing generalize well to unseen subjects.
Effect of Number of Selected Ex- perts: Similarly, Figure demon- strates that an increase in the number of selected experts generally enhances performance. However, this improve- ment comes at the expense of longer inference times. As the number of se- lected experts increases, the number of parameters required during infer- ence also rises, leading increased
Table I: Performance analysis of the model while training with and without proxy loss with varying number of selected experts.
k=2
3
K=4
K=5
K=6
No Leroxy
62,49
64.15
67.32
67.86
66.21
Pymxy
62.10
65.37
68.93
68.57
67.03
It should be noted here that, given the number of experimental runs required to perform this analysis we only indicate the results after 10th epoch in this analysis, which may not reflect the best performance of the model.

---

Table 3: Verification Performance (TAR (%) @FAR=%) for face included treatment and control protocols of the BTS 3.1 dataset. All faces are detected and algined using RetinaFace face detector.
Face Included Treatment
Face Included Control
Feature
Dataset
10
10
10
10"
10
10
10
10
GAP
11
Adaface 8
Briar
76.6
58.4
43.3
32.1
98.5
94.6
88.9
81.2
NAN
20
Adaface [8]
Briar
78.5
61.2
46.8
33,4
98.5
95.3
89.3
84.8
MCN
19
Adaface g
Briar
79.4
62.9
47.3
35,9
98.5
95.9
90.7
85.7
CoNAN [5]
Adaface
8
Briar
81.3
64.3
49.6
36.8
98.6
96.2
91.8
86.1
ProxyFusion
Adaface
g
Briar
83.7
68.9
53.9
40.1
98.6
96.8
92.7
88.3
inference time. Our observations indicate that setting the number of selected experts, K, to achieves performance close to the best model while significantly reducing computation time.
Contribution of Proxy Loss Proxy: In Table [ we present the contribution of the Lroox to the overall performance. We observe that though the contribution of Lrooxy marginal when the number of selected experts is small, there are performance improvements at modest N. We believe this is due the decorrelation of proxies, which results in extraction of diverse information from the feature set.
3.4 Inference Time and Computational Cost:
Comparison GFLOPs for Different Methods
To validate our algorithm's claimed linear time complexity perform GFLOPs analysis of our method against increasing sizes of the feature- set. As can be observed from Fig. S we start with small feature-se set size of 100 and scale it up to I million. Our fusion model's (in blue) GLOPs in- creases linearly with the increasing number of features the set. con- trary, GFL.Ps increase quadratical for models like CoNAN [5] and [9]. Moreover, these with single-shot in- put of large feature-sets to these mod- els, the memory footprint increases quadratically due to intra-set attention which leads out-of-memory issues for these models beyond N of 21000. These experiments performed on NVIDIA A6000 averaged over runs.
0.0
Meen
644369
lagrericn
-1.5
0100572
0.00201
6.00132
2000020000
50000020303
1000000
Figure 5: Time complexity comparison of ProxyFusion ap- proach against SoTA. On the Y-axis we plot the Log of GFLOPs with base 10, and axis the number of features in the feature set N.
Table 2: Rank-1 accuracy (%) for video-wise iden- tification on DroneSURF dataset.
3.5 Comparison to SotA methods
In table we compare our results with the state- of-the-art (SOTA) in face verifcation using the RetinaFace face detector and Adaface Features on BTS3.1 Dataset. In table 4 we present re- sults using MTCNN face detector and Adaface and Arcface feature extractors on BTS3.I. As
Trained On DroneSURF
Feature
Active
Passive
GAP 11
Adaface 8
46.87
7.29
NAN 20
Adaface
8
65.62
6,25
MCN [19
Adaface
8
72.92
8.33
mentioned earlier, for fair comparison to meth- ods such as [9] in first half of table 4 we train our model on WebFace dataset using the pre- computed Adaface features released by [9].
CoNAN 5
Adaface 8
80.21
13.54
Ours
Adaface 8
83.33
13.54
Trained On BRS (Cross-Datae Dataset Evaluation)
GAP [11
Adaface 8
46.87
7.29
As can observed from table and our method significantly outperforms other linear time methods such as NAN 20 and MCN while providing significant jump over naive
NAN 20
Adaface 8
80.21
8.33
MCN 19
Adaface
8
79.16
10.41
CoNAN 15
Adaface
8
83.33
12.50
Ours
Adaface
8
80.21
13.54

---

Table 4: Verification Performance (TAR (%) @FAR=%) for face included treatment and control protocols of the BTS 3.1 dataset. All faces are detected and algined using MTCNN face detector.
Face Inel.
Trt.
Face
Incl. Ctrl.
Feature
Dataset
Time
No Inter.
10
10
10-1
10
10
10-
GAP [1]
Adaface
WF4M
((N)
50.8
40.8
31.7
91.3
86.9
80.1
CAFace 9
Adaface
WF4M
O(N*)
51.3
42.0
33.4
92.7
88.8
83.0
ProxyFusion
Adaface
WF4M
O(N)
4
51.5
41.7
33.69
92.9
89.2
83.2
GAP 11
Arcface
3
Briar
((N)
37.0
27.3
19.5
84.8
75.3
66.2
NAN 20
Arcface 3
Briar
O(N)
<
39.0
26.6
18.3
84.3
72.9
60.4
MCN 19
Arcface 3
Briar
((N)
V
39.4
28.2
19.4
87.1
77.9
67.2
CoNAN [5]
Arcface [3
Briar
((N=)
43.4
32.1
23.1
87.6
81.0
71.9
ProxyFusion
Arcface
Briar
((N)
42.1
31.4
23.4
88.4
81.8
73.9
Inter, denotes whether the method depends on intermediate features for feature fusion.
If method requires intermediate features, it is not compatible with legacy templates.
averaging or Global Average Pooling (GAP) 11 Our method also achieves on-par performance to quadratic time complexity methods such CoNAN (5) and CAFace [9]. When trained Web- Face, our approach outperforms CAFace 9 on most FAR thresholds without utilzing intermediate features unlike [9] When compared CoNAN (5). our method performs outperforms it Face included control setting while being significantly faster (See Fig. 5] having much lower runtime memory requirements, and comparable inference time parameters (T3.6M for CoNAN and 12.4M for ProxyFusion). Similar trend can be observed on DroneSURF dataset (refer Table All SotA experiments are performed several times and we present the average results. We calculated Standard Errors of the Mean (SEM) across the multiple observations and found the SEM to very low in the range of 0.008-0.09 showing the statistical significance of the experiments.
Related Work
Methods based on Intermediate Feature Maps and Metadata: Leveraging intermediate feature maps for feature fusion, methods like the nonlocal neural network [18] and RSA [12) show promise in modeling intra-set relationships through detailed spatial analysis. These methods utilize intermediate feature maps U: of size C... H x W capture complementary information and refine spatial relationships. CAFace [9] addresses the computational issue of these approaches with a two-stage feature aggregation method, using these feature maps as style information. This involves assigning N inputs to MM global cluster centers and fusing clustered features. CAFace generates an affinity map with 2 complexity [15]. The requirement for intermediate feature maps makes these methods incompatible with legacy biometric systems lacking precomputed intermediate features. In contrast, our method relies solely on the penultimate feature representation, typically used for generating biometric templates. Furthermore, Metadata extracted from face images can serve as an alternative source for inferring face quality. (e.g., yaw, pitch, face size) and utilizes a siamese network to determine the relative quality correlations among face images in a set. Additionally, metadata can facilitate fine-grained matching across galleries and probes by identifying faces with similar characteristics. For instance, TADPool [33 adapted both probe and gallery features by considering attributes like face yaw and roll, employing an attention block to pool probe features based on their compatibility with selected gallery features.
[4 introduced feature aggregation method that integrates metadata
Linear Time Feature Fusion Methods: Previous works have proposed aggregation methods with linear time complexity against the number of features in set. In [20), a network architecture with two cascaded attention blocks is presented, which evaluates the significance of features an image set and uses these scores for feature aggregation. Recently, [15] employed a differentiable coreset selection approach, using learned metrics and a Gunbel-Softmax distribution to optimize the selection of a small, representative subset, which is then enriched via self and cross-attention mechanisms thereby reducing computational complexity by decreasing set-size. [19] introduces a multicolumn network that assigns weights to images within a set based on their visual quality, assessed through self-quality assessment module. This network then dynamically adjusts these weights according to each image's 's relative content quality compared to others in the set.

---

Conclusion
We address key challenges in feature fusion: (i) real-time inference, (ii) cross-distribution matching, (iii) generalization large feature sets, and (iv) compatibility with legacy feature stores. Our proxy-relevancy based approach for expert selection, combined with a feature pooling approach, ensures robust multifaceted feature aggregation with minimal inference time and no need for extra metadata or intermediate features. Ablation studies demonstrate our method's superior performance and efficiency, outpacing existing approaches in both speed and effectiveness for real-time upplications.
Potential Negative Societal Impacts: We are meticulous about training or testing our models only on datasets with approved IRB and consent from involved human subjects. This is why we do not use LJBBB/C datasets, as they contain web-sourced faces without consent or IRB approval. Conversely, datasets like IARPA BRIAR and [ which we utilize are collected under IRB with subject consent. Limitations: While our method effectively discerns feature informativeness via sparse experts, relying solely on feature-set distribution statistics overlooks fine aeeeggraind intra-set relationships. We can enhance this by hierarchically merging them with intra-set methods, achieving further gains with limited computational cost.
Acknowledgments
This research based upon work supported in part by the Office of the Director of National Intelli- gence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via 2022-21 1011100100000 The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
This works is also partially supported by the the National AI Research Institutes program by the National Science Foundation (NSF) and the Institute of Education Sciences (IES), U.S. Department of Education, through Award #2229873. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the NSF, the IES, or the U.S. Department of Education.
References
[1] David Cornett, Joel Brogan, Nell Barber, Deniz Aykac, Seth Baird, Nicholas Burchfield, Carl Dukes, Andrew Duncan, Regina Ferrell, Jim Goddard, et al. Expanding accurate person recognition to new altitudes and ranges: The briar dataset. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 593-602, 2023.
[2] Jiankang Deng, Jia Guo, Evangelos Ververas, Irene Kotsia, and Stefanos Zafeiriou. Retinaface: Single-shor shot multi-level face localisation in the wild. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2202-5211 2020.
[3] Jiankang Deng. Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[4] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1-39, 2022.
[5] Bhavin Jawade, Deen Dayal Mohan, Dennis Fedorishin, Srirangaraj Setlur, and Venu Govin- daraju. Conan: Conditional neural aggregation network for unconstrained face feature fusion. In 2023 IEEE International Joins Conference on Biometrics (IJC)) pages 1-9, 2023.
[6] Isha Kalra, Maneet Singh, Shruti Nagpal, Richa Singh, Mayank Vatsa, and P. B. Sujit. Dronesurf: Benchmark dataset for brone- face recognition. In 2019 14th IEEE International Confer- ence on Automatic Face and Gesture Recognition (FG 2019), pages 2019.
10

---

[7] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 33:1 83:16666188773 2020.
[8] Minchul Kim, Anil K. Jain, and Xiaoming Liu. Adaface: Quality adaptive margin for face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18750-18755 June 2022.
[9] Minchul Kim, Feng Liu, Anil K Jain, and Xiaoming Liu. Cluster and aggregate: Face recognition with large probe set. Advances in Neural Information Processing Systems, 35:36054-36066, 2022.
[10] Minchul Kim, Yiyang Su, Feng Liu, Anil Jain, and Xiaoming Liu. Keypoint relative position encoding for face recognition. arXiv preprint arXiv:2403.1 14852, 2024.
[11] Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. arXiv preprint arXiv: 1312,4400, 2013.
[12] Xiaofeng Liu, Zhenhua Guo, Site Li, Lingsheng Kong, Ping Jia, Jane You, and BVK Kumar. Permutationninvariiat feature restructuring for correlation-awan image set-based recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4986-4996, 2019.
[13] Nishant Sankaran, Deen Dayal Mohan, Sergey Tulyakov, Srirangaraj Setlur, and Venugopal Govindaraju. Tadpool: Target adaptive pooling for set based face recognition. In 202/ 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021), pages 1-8, 2021.
[14] Nishant Sankaran, Sergey Tulyakov, Srirangaraj Setlur, and Venu Govindaraju. Metadata- based feature aggregation network for face recognition. In 2018 International Conference on Biometrics (ICB), pages 1 118-123, 2018.
[15] Gil Shapira and Yosi Keller. Facecoresetnet: Differentiable coresets for face set recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 7744-4756, 2024.
[16] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experrs layer. arXiv preprini arXiv: /701. /et.:/1011..... 2017.
[17] Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018
[18] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2018
[19] Weidi Xie and Andrew Zisserman. Multicolumn networks for face recognition. arXiv preprint 1807.00 1922, 2018.
[20] Jiaolong Yang, Peiran Ren, Dongqing Zhang, Dong Chen, Fang Wen, Hongdong Li, and Gang Hua. Neural aggregation network for video face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4362-4371, 2017.
[21] Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, and Yu Qiao. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE signal processing letters, 23(10): 1499- 1503, 2016.
[22] Zheng Zhu, Guan Huang, Jiankang Deng, Yun Ye, Junjie Huang, Xinze Chen, Jiagang Zhu, Tian Yang, Dalong Du, Jiwen Lu, et al. Webface260m: A benchmark for million-scale deep face recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We clearly state the contributions, and our paper is about these contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Yes, discuss limitations in the conclusion.
12

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We include all of the information necessary to reproduce our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
13

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We will publicly release the code and trained models. The data can be requested from IARPA and IIT Jodhpur.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips.cc/public/eee. www.ddddeeeooeuuummmceco ssi ssionyolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
14

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide all training and evaluation details in the paper.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Comparison to baseline methods like mean and state of the methods provides information about the statistical relevance of our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-simma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We include these details in the paper.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
15

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http::ss////riis.... htt.t//iiiiiir//.a.......ooooooa
Answer: [Yes]
Justification:
We follow all of the guidelines in the NeurIPS code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have included this discussion in our paper.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification: There are particular safe guards, yet we only train our models on datasets with approved IRB and consent from involved subjects. Therefore there is limited risk of misuse of our pretrained models.
16

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We explicitly state in what existing assets we using, and properly attribute them.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pacerswithcode thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors are encouraged to reach out the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We document the new assets we have created and we will release the documents with the assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
17

---

Answer:
[No]
Justification: The datasets we are using are provided external entities. We sign required agreements and licenses to acquire these datasets.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: Yes, all of the datasets using have IRB approvals. See potential negative societal impact section our conclusion.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
18


Reviews:
[{"review_id": "b9w6YO1FWh", "reviewer": "Reviewer_vG52", "summary": "The authors propose a method for fusing face features extracted from sets of images.\nSeveral proxies are defined that are trained to be specialized for certain aspects of faces, and the most relevant for a given set of face images are obtained.\nImportance weights for different samples are estimated for each selected proxy, and a template stores a combination of weighted feature averages.\nExperiments on challenging datasets with a wide variety of viewing conditions indicate largely improved performance with linear time complexity in the number of samples per set.", "questions": "The most important question is how negative samples are obtained for the proxy loss. The authors should elaborate on this. Also, the strategy on handling several features per template needs to be revealed. Afterward, I am happy to increase my rating.", "limitations": "The authors took special care about using only datasets with IRB-consented data. They should be aware, however, that the pre-trained networks that they exploit for feature extraction have been trained on web-scraped datasets which did not include IRB-approved subjects.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "+ The paper is written well, and the motivation of the proposed method is clearly stated.\n+ The results show improved performance over state-of-the-art feature fusion approaches on difficult datasets.\n+ Ablation studies show the importance of most parts of the pipeline, and provide insights on relevant parameters of the model.\n+ Besides a few minor mistakes, there are no apparent errors in the math.", "weaknesses": "1. There are a few important details of the proposed algorithm missing in the paper:\n\n   a) One part of the proposed method, i.e., the details of the expert network E_j are missing in the paper -- how large are the layers? \n\n   b) The proxy loss is defined over a set of negative samples. There is no word spent on how to select such negative samples -- yet, the selection is of high importance. Also, this loss includes a lambda value that is not detailed.\n\n   c) Both gallery and probe templates finally consists of a concatenation of several averaged features. The authors need to clarify how the final score between these sets of templates is computed.\n\n\n2. Some parts of the paper include non-standard solutions that should be considered to update:\n\n   a) In order to compute valuable means and standard deviations in section 2.2, the features would need to reside in Euclidean space. However, modern deep FR networks optimize for cosine space, and deep features typically do not follow Euclidean distributions. Why would this be a good idea?\n\n   b) Relatedly, the expert networks obtain mean, standard deviation and proxy centers. It is not shown in the paper that the variances provide any improvements over only using the means. A better handling of mean and proxy center than a simple concatenation is likely possible.\n\n   c) The proxy center projections v are selected to be equidistant in Euclidean space, while they result from a projection that rather lives in cosine space. It is not clear where the expression is coming from or why such a difficult expression needs to be defined, a simple set of K-dimensional basis vectors e_i would likely be sufficient.\n\n   d) The models used for feature extraction are not well-suited for low-resolution and highly distorted faces. Better models will likely improve the utility of the feature fusion approach.\n\n\n3. There are some improvements on the paper:\n\n   a) The introduction is slightly repetitive and could be shortened.\n\n   b) At several places in the paper, it is mentioned that the method is compatible with existing template storage solutions. The practical relevance for such compatibility is questionable. In a typical template storage, a single template is stored as an aggregation over several samples already. The authors should not overemphasize this characteristic of their method.\n\n   c) The authors claim their algorithm to have a time complexity of O(N). However, the complexity is rather O(N*K). Depending on the choice of K, this might be a relevant deviation from O(N).\n\n   d) In line 138 and following, the authors write about batch creation strategies. It is not entirely clear if this is applied during test time as well, or whether this is a training time issue only. Finally, the authors claim to include diverse features, but random selection does not ensure diversity.\n\n   e) The equation for the standard deviation in section 2.2 is slightly off, the factor should be 1/(N-1).\n\n   f) In line 170, the scores s_ik should make use of the projected proxies Wp, not the proxies themselves.\n\n   g) The figures should be included as vector graphics to allow zooming. Especially figure 3 is of too low quality to read.\n\n   h) In the caption of figure 5, N represents the total number of probe features. In section 2.2, N is the number of images for one identity, while in section 2.3, N represents the number of identities.\n\n   i) The paper should be carefully checked for spelling mistakes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper and providing constructive feedback! Below we provide the answers to your questions. The amends made based on your suggestion and feedback will reflect in the final-version. \n\n1.a) We have provided brief implementation details about the expert networks at line 214 in section 3.2. For the expert networks we utilize a three layer MLP with LeakyReLU activation and a dropout with probability of 0.5. More specifically, the first layer in the MLP projects from `512 * 3` to `512 * 2`, the second layer from `512 * 2` to `512 * 2`, and last layer from `512 *2` to `512`. Overall the MLP has 3.14M learnable parameters.\n\n1.b) Thanks for asking about **negatives in proxy loss**. Below we provide an explanation for the same which will also be added to the paper. \n\nOur loss consists of two components: (i) Identity loss and (ii) Proxy loss. The proxy loss aims to ensure that during training, proxies capture unique and mutually exclusive latent facial information, aiding in the selection of specialized experts for different facial traits. To guarantee that proxies are maximally distinct, we start with K equidistant standard basis vectors (equal to the number of proxies). Our objective requires proxies to converge towards their respective vectors and diverge from all others. This is achieved by optimizing the following :\n\n$L_{\\text{Proxy}} = \\frac{1}{K} \\sum_{i=1}^K \\left[ \\ln \\left( 1 + \\exp \\left( -\\alpha (s_{ii} - \\lambda) \\right) \\right) + \\frac{1}{|K-1|} \\sum_{k \\in K \\ \\text{and} \\ k \\neq i} \\ln \\left( 1 + \\exp \\left( \\beta (s_{ik} - \\lambda) \\right) \\right) \\right]$\n\nWhere, $s_{ik} = \\left( p_i \\cdot v_k \\right) / \\left( {\\|p_i\\| \\|v_k\\|}\\right)$ is the similarity between the $i^{th}$ proxy and $k^{th}$ fixed basis vector. The first part of this term enforces that proxy goes closer to its respective basis vector, while second part enforces that it goes away from all other negative basis vectors. To answer your question, the negatives in this proxy loss are these negative basis vectors (basis vectors not corresponding to the concerned proxy). $\\lambda$ is a threshold which is set 0.1. Here $\\alpha$ and $\\beta$ are the scaling parameters for the positive and negative similarities respectively.\n\n1.c) As stated in line 131 - We define the final template vector t as a concatenation of $s_j$ given by: \n\n$t = \\left[ s_1, s_2, \\ldots, s_\\widehat{K} \\right]$\n\nleading to a single feature representation (template vector) for each gallery and probe respectively. The final scores between the templates (gallery and probes) are computed by calculating the similarity between these vectors.\n\n2.a) When calculating the mean and variance of feature sets from deep face recognizers for expert networks, we use un-normalized feature vectors. These vectors, not lying on a hypersphere and following a Euclidean distribution, are only unit-normalized before computing cosine similarities. This will be clearly stated in the paper and relevant sections.\n\n2.b) Mean and variance summarize the distribution of the feature-set. The mean indicates the center, and the variance describes the spread. This is essential, as also noted in previous studies like CoNAN [5], which also examines different statistical measures used for conditioning.\n\n2.c) We do not enforce proxy-centers to be equidistant in euclidean space. We enforce them to be equidistant in the cosine-space (unit-hypersphere). We agree with your point that the statement is rather complex and we will simplify that for the final-version.\n\n2.d) We absolutely agree with your comment. To be fair and consistent to prior methods for comparison, we utilize the same backbones as used by CAFace [8] and CoNAN [5].\n\n3.a) Thanks for the great suggestion, we have shortened our introduction to not be repetitive. This will be reflected in the final-version.\n\n3.b) We acknowledge the reviewer's point and will accordingly reduce the emphasis as suggested. In law enforcement applications of surveillance and biometrics, such as IARPA's JANUS and BRIAR programs, systems utilize set-to-set template matching that maintains non-aggregated sample-level features in biometric stores. Our proposed approach aligns with these legacy template stores, and we will clarify this further in the paper. Thank you for the valuable feedback.\n\n3.c) Depending on the choice of $\\hat{K}$ (Number of selected experts), the complexity might deviate from O(N). Including model hyperparameter $\\hat{K}$, the overall complexity is $O(N*\\hat{K})$. But once trained, $\\hat{K}$ is constant and typically very small ($\\hat{K}$ <<< N; we use $\\hat{K}$ = 4 regardless of dataset, with N ranging to thousands of features per video). Thus, with respect to the number of incoming features, our model has a linear complexity O(N).\n\n3.d) This is only used during training to augment the feature set with samples from multiple videos of a subject. While random selection may not be optimal for diversity, it often performs well in deep learning model training. This is not applied during test time.\n\n3.e, f, g, h and i) Thanks for providing these suggestion and pointing out the required corrections. We have made the appropriate amends in the paper for final version and have performed a thorough spelling and grammar check.\n\n**The strategy on handling several features per template needs to be revealed:** Our method handles varying number of features per template through the use of mean-and-variance as the conditioning information for the experts. Since the final aggregation scores per expert are computed through the use of similarities (referred to as divergence scores) between the original features and the set-centers the overall method can handle varying number of features per template. Furthermore, during batch-creation, we randomly sample features from multiple videos for an identity to create diverse feature-sets of varying lengths, this further helps in models generalizability.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cOF64CAL5f", "reviewer": "Reviewer_waA1", "summary": "The authors introduce a novel approach for face feature fusion, addressing typical scenarios such as low-quality probes and high-quality or different domain gallery sets of faces. They describe alternative approaches and conclude that these are currently limited because they are either not compatible with legacy biometric databases or have real-time performance issues.\n\nThe authors provide a method for feature aggregation through Sparse Experts. The overall pipeline includes three steps: Feature Extraction -> Expert Network Selection -> Sparse Expert Network Feature Aggregation. Each step is described in detail.\n\nOptimization is done using a two-component loss: Loss for the identification objective + proxy loss. The proxy loss aims to make different proxies attend to different facial characteristics. The authors present experimental justifications for the second loss in the experimental section.\n\nThe authors describe the experiments conducted. Two datasets were used for training: BRIAR Research Set 3 and WebFace 4M. Evaluation was performed on BTS 3.1 and DroneSURF. The authors provide all details about these steps. They also present visualizations showing that each expert assigns higher weights to faces with more discriminative identity information and minimal weights to poor-quality images. Experimentally, they found that the optimal setup is to use 4 Selected Experts and 12 total Proxies/Experts.\n\nThe authors compared their method with various state-of-the-art alternatives and demonstrated that their method either produces strong results, achieving SOTA performance, or performs on par with quadratic-time methods, despite being a linear-time method.\n\nAdditionally, the authors justify that their method is feature order and set length invariant.", "questions": "1. Could the authors provide the code and necessary weights at this review stage to enhance clarity and reproducibility?\n2. Could the authors provide more detailed information about the DroneSURF random splits or share the splits themselves?\n3. Could the authors clarify about the statistical significance?", "limitations": "There is one limitation described:\n\n> While our method effectively discerns feature informativeness via sparse experts, relying solely on feature-set distribution statistics overlooks fine-grained intra-set relationships.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The idea of the method is quite clear. The algorithm has a linear time complexity, while alternatives with similar quality have quadratic time complexity. The method is invariant to set length and feature order. It can be applied even to legacy databases, which is highly important.\nThe paper is clearly written, and the results seem to be very promising. The visualizations are well drawn, clear, and insightful.", "weaknesses": "The authors commit to releasing the code but did not attach it. While the method is clear and straightforward, the overall implementation can be complex, which could hinder reproducibility. Therefore, I believe the code is a crucial part of this work and should be shared along with the article, especially since this work does not involve heavy neural models or long training procedures.\n\nWithout a full code review and the ability to run it, there may not be sufficient proof of reproducibility. Additionally, the authors' response to Question 7, 'Experiment Statistical Significance,' may not fully address the issue, as comparison with alternative methods alone is not evidence of statistical significance.\n\nFurthermore:\n\n> (ii) DroneSURF: This dataset includes Active and Passive Surveillance settings, each with 100 videos and 786,000+ face annotations. Following [12], we split subjects randomly: 60% (34 identities) for training/validation, 40% (24 identities) for testing. The dataset has 200 videos of 58 subjects, over 411,000 drone-captured frames. Results are based on the 203 video-wise identification protocol.\n\nIt is unclear how the random split can be compared to other methods. If the split was done as in [12], this information should have been provided. Additionally, I suggest that the splits be released so future authors can make clear comparisons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Reviewer Comments**\n\nWe appreciate the thoughtful critique and the opportunity to enhance the clarity and reproducibility of our work. Below are our responses to the concerns raised:\n\n1. **Code Availability and Implementation Details:**\n   Thank you for emphasizing the importance of code availability for reproducibility. We have now made the code publicly available on an anonymous GitHub repository to facilitate the review process. It can be accessed at [https://github.com/anonymousdoubleblindreview/ProxyFusion](https://github.com/anonymousdoubleblindreview/ProxyFusion), complete with instructions for installing the necessary requirements. Please note that the datasets used for training and evaluation are externally sourced, and access requests should be directed to the original authors of those datasets as detailed in our repository links.\n\n2. **Statistical Significance of Experiments:**\n   We acknowledge the need for a rigorous demonstration of statistical significance and have addressed this issue as follows:\n   - **Error Margins:** We conducted our experiments ten times, presenting results alongside their corresponding Standard Errors of the Mean (SEM), calculated as \n$$\n\\(\\text{SEM} = \\frac{s}{\\sqrt{n}}\\)\n$$\n, where $\\(s\\)$ is the standard deviation and $n$ is number of observations. The low SEM values as can be observed in our results table below suggest minimal deviation/variance in model performance across multiple runs. The results below are presented on BTS3.1 dataset with Adaface feature extractor, Retinaface face-detector and our proposed fusion strategy - ProxyFusion. We will calculate and incorporate SEM values for all tables in the paper for statistical significance. \n\nFor Face Included Treatment\n| Method       | Feature | Dataset | 10−1   | 10−2   | 10−3   | 10−4 |\n|--------------|---------|---------|--------|--------|--------|--------|\n| ProxyFusion  | Adaface | Briar   | 83.7   | 68.9   | 53.9   | 40.1   |\n| **SEM**      |         |         | 0.043 | 0.072 | 0.076 | 0.091 | 0.108 | \n\nFor Face Included Control\n| Method       | Feature | Dataset | 10−1   | 10−2   | 10−3   | 10−4 |\n|--------------|---------|---------|--------|--------|--------|--------|\n| ProxyFusion  | Adaface | Briar   | 98.6   | 96.8   | 92.7   | 88.3   |\n| **SEM**      |         |         |  0.008 | 0.012 | 0.037 | 0.071 |\n\n\n   - **Comparison with State-of-the-Art (SoTA):** Our method consistently outperforms previous SoTA methods by 2% across all False Acceptance Rate (FAR) thresholds. This improvement is in line with the incremental gains typically observed in this research domain, affirming the significance of our contributions.\n   - **Computational Complexity:** It's pertinent to highlight that our approach achieves these improvements with an O(N) complexity, unlike some earlier $O(N^2)$ methods like CAFace [8] and CoNAN [5]. This efficiency further validates the practicality and significance of our findings.\n\n3. **Details on the DroneSURF Dataset Splits:**\n   Regarding the DroneSURF dataset, we understand the necessity for standard protocol to enable fair comparisons. In the paper we had stated the DroneSURF matching protocol as described by the original authors of the dataset in [6]. Moreover to be consistent with the methods used by previous works such as CoNAN [5], we used first 24 identities (40%) for testing and remaining 64 identities (60%) for training based on their subjectID order. We will include the protocol and share the split in our repository to assist future researchers in replicating and comparing results accurately.\n\nWe hope that these updates adequately address your concerns and enhance the comprehensibility and reproducibility of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2j1NgIfhrm", "reviewer": "Reviewer_BueQ", "summary": "The paper proposes a novel feature fusion technique for unconstrained face recognition via sparse experts. The authors propose a expert network selection mechanism using the $k$ proxies, $p_j$, and all the $N$ precomputed face features, $f_i$. The top $\\hat{k}$ selected experts are used by the 3d set centers to produce discriminative features per expert that finally forms the template. Two losses are proposed, a supervised contrastive loss ($L_{id}$) to ensure high interclass and low intra class separability between probe and gallery, and a proxy loss ($L_{proxy}$) to ensure the experts learn mutually exclusive information. The authors validate their method primarily on DroneSurf and BRIAR dataset on different TAR@FAR=thresholds. They also provide visualization for the learned experts and GFLOPS analysis to support the linear time complexity claim.", "questions": "I have mentioned most of the questions in the weakness section. The most important ones to address would be 1 and 2.", "limitations": "The authors adequately address the limitations about the proposed methods and use of only IRB approved datasets for experiementation. \n\nFor future work, it would be interesting to compare the method against [1], which also attempts to achieve similar goals. This however, is a suggestion, and in no way affects my final rating for this conference.\n\n\n[1] Shapira, Gil, and Yosi Keller. \"FaceCoresetNet: Differentiable Coresets for Face Set Recognition.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 5. 2024.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The application of mixture of experts acts as a way of diffentiable sampling to select the right experts and intrinsically, the right feature for aggregation. \n* The paper presents a strong evidence emperically, especially with the BRIAR dataset, which is a highly noisy and a challenging dataset. \n* The authors detail the contribution on all aspects that includes - interpretability of weights for set-centers, effects of number of proxies, effects of expert selection, and finally, the proxy loss.", "weaknesses": "### 1. The presentation of the method end experiment section could be greatly improved. There are some texts that is unclear that certainly affects the final rating of the paper. \n\na.  Line 97, What are proxies in this context and how are they constructed initially? Are they randomly initialized? Or are they some form of router network? The paper also seems to use proxies/learnable queries (line 68) interchangeably which is further confusing.\n\nb.  Line 125, Divergences represent the dissimilarity between two distributions. Can the authors please expand on how the dot product between two feature vector is a divergence?\n\nc.  Table 1, What dataset has this been tested on? Is this where the authors employed $\\hat{K}=4$ as the optimum experts?\n\nd.  What is the success rate of both face detectors for probe images? At 500m with several noise factors, these standard detectors might encounter higher degree of failure.\n\ne.  Figure 4 is a bit confusing as to what exactly $x-axis$ is, as the experts/proxies are used interchangeably.\n\nf.  It would also add value if authors are able to add TAR@FAR values per each distance category for probe BTS 3.1. I am assuming this would involved categorizing the results which should not be a completely new experiment.\n\n\n### 2. The paper does not include enough description to make it reproducible.\n\na.  Line  141, what were the chosen $L$ and $U$ values in the experiments?\n\nb.  Line 143, what was the suitable $M$ found to be?\n\nc.  Line 146, pad them with what value?\n\nd.  What is the final $d$ and $d'$ ?\n\n\n### 3. I also have few minor corrections to suggest to further improve the quality of the paper.\n\na.  Paragraph 139 and 143 seems to convey the same meaning. Can be combined into one.\n\nb. SCL equation for $L_{id}$ uses the transpose $T$ inconsistently.\n\nc. Line 187, I think it is supposed to be \"For training *ProxyFusion*...\"\n\nd. Other errors:\n\n * Typo in line 90, *refere*\n* Line 125, 'that is used to compute...'\n* Line 154, \"primary goal bridge....\"\n* Line 282, \"these with single shot...\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your valuable feedback. We have addressed your questions below. \n\n1.a) We define proxies as learnable embeddings / vectors of dimension length 512 (same dimensionality as the face-feature embeddings). These embeddings represent latent information about facial characteristics required to decide which expert network should be utilized. We initialize proxies using kaming-he initialization, and then enforce them to be farthest apart using our proposed proxy-loss. Thanks for suggesting to not use learnable queries and proxies interchangeably as in line 68. We will correct this in camera-ready.\n\n1.b) We compute the similarity of the original input features to the set-centers extracted from the selected expert-networks. These similarity scores tell us how much is an original feature diverging (or dissimilar) from the set-center - smaller the similarity, more diverging/dissimilar is the feature from the set-center. Hence we utilize the term divergence scores to describe these scores. These are then softmax-normalized and utilized as weightages for feature aggregation.\n\n1.c) Thanks for noticing this. Table 1 results are on BTS3.1 dataset, face-included treatment setting with Adaface feature extractor. We will provide this information explicitly in the camera-ready paper. The optimum value of $\\hat{K}$ = 4, can be discerned from Figure 4 and Table 1. Figure 4 shows that at $\\hat{K}$=4 number of selected experts and 11 number of total experts, the model reaches closest to best performance. The ablation is present on the test-set, the selection of $\\hat{K}$=4 was performed on a subject disjoint validation-set.\n\n1.d) We utilize standard detectors, specifically MTCNN and Retinaface in this work for fair comparison to previous works such as CAFace [8], CoNAN [5] etc. Performance of the face-detectors at long distances such as 500m is not directly benchmarked in previous works due to lack of human-annotated ground truths. Change of face-detectors significantly impacts the overall recognition performance as evident from Table 2 and Table 3 since better face detectors such as Retinaface detect more number of faces in a video leading to better recognition performance post fusion.\n\n1.e) Total number of proxies and total number of experts are always equal, since proxies help in selecting the right expert for a given feature-set. X axis on Figure 4 denotes this “total” number of proxies (or total number of experts since they are equal). Y axis in Figure 4 denotes the number of selected top-k experts. This is always less than the total number of experts. In summary Figure 4 presents an ablation on the performance of the model as the total-number of experts and number of selected top-k experts change. We will make this more clearer in the camera-ready.\n\n1.f) Thanks for the great suggestion. Below is one set of segregated results based on distance. We will provide more detailed analysis in the final-version.\n\n| Distance | Method       | TAR@FAR=10^-1 | TAR@FAR=10^-2 | TAR@FAR=10^-3 | TAR@FAR=10^-4 |\n|----------|--------------|---------------|---------------|---------------|---------------|\n| 500m     | GAP          | 46.72%        | 36.31%        | 29.22%        | 23.75%        |\n| 500m     | **ProxyFusion** | **48.57%**    | **38.94%**    | **31.84%**    | **25.91%**    |\n| 200m     | GAP          | 62.13%        | 51.20%        | 44.53%        | 36.27%        |\n| 200m     | **ProxyFusion** | **62.93%**    | **53.87%**    | **46.67%**    | **37.87%**    |\n| 100m     | GAP          | 69.27%        | 57.80%        | 48.41%        | 39.17%        |\n| 100m     | **ProxyFusion** | **70.70%**    | **60.83%**    | **51.27%**    | **41.72%**    |\n| <10m     | GAP          | 70.74%        | 62.38%        | 55.82%        | 48.05%        |\n| <10m      | **ProxyFusion** | **71.85%**    | **64.79%**    | **58.48%**    | **50.90%**    |\n\n2.a) Thanks for bringing this to our notice. The chosen value of L is 100 and U is 1200 during training.\n\n2.b) M denotes the number of identities sampled to create one batch. We utilize M as 170 based on our GPU’s memory capacity. \n\n2.c) We pad with “zero-vectors” of dimensionality D to match the length of the biggest feature-set in the batch. This is just performed to create uniform shape tensors for training. \n\n2.d) The value of d is 512, and d’ is 10.\n\n3.a) Thanks for noticing this. We will combine the paragraphs into one for camera-ready.\n\n3.b) Thanks for noticing this. We will add transpose to the denominator. \n\n3.c) Thanks for noticing this. We will make the correction.\n\n3.d) We will thoroughly amend the paper for typographical and grammatical errors. Thanks for your comments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LBjhyTds3R", "reviewer": "Reviewer_Cxak", "summary": "This paper presents a novel face recognition framework that aims to address the challenges of feature fusion in long-range, low-resolution scenes. The authors propose a linear time complexity approach that is compatible with traditional biometric template databases and does not require additional metadata or intermediate feature maps.ProxyFusion utilises a set of learnable agents to implicitly represent potential facial attributes and selects the most relevant experts to focus on the feature set, thus significantly reducing inference time and parameters. The method is order and size invariant, making it very robust for real-time applications and large probe sets. Extensive experiments on the IARPA BRIAR BTS3.1 and DroneSURF datasets demonstrate the superiority of ProxyFusion in unconstrained long-range face recognition settings.", "questions": "Please refer to the weaknesses.", "limitations": "Please refer to the weaknesses.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. In this paper, the authors propose a novel feature aggregation framework to address key challenges in face recognition, especially in low-resolution and long-range scenes. \n2. The runtime of ProxyFusion's feature aggregation method is linearly related to the number of features, which is very effective for real-time reasoning and processing large probe sets.", "weaknesses": "1. The optimization of the sparse experts is ambiguous. In the section of methods, the authors do not provide any details on how to optimize the sparse experts. \n2. The paper mentions that relying solely on feature-set distribution statistics might overlook fine-grained intra-set relationships. This suggests that the model may not capture all the nuances present within a set of features.\n3. The performance of ProxyFusion, like any face recognition system, is likely to depend heavily on the quality of the input data. Poor quality inputs could degrade performance.\n4. The paper does not mention the model's robustness to adversarial attacks, which is an important consideration in the field of biometrics and face recognition.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. **Optimization:** Thank you for your feedback. For clarity on the optimization of the sparse experts, please refer to section 2.3 titled \"Optimization\" in our paper. There, we elaborate on the overall optimization objective. Here's a more detailed explanation of our optimization strategy.\nOur loss has two terms: \n\n> **Identity Loss**: This component ensures that the aggregated embeddings from the same identity—collected under varying qualities—are drawn closer, while distancing them from the embeddings of different identities. We implement this through a supervised contrastive loss.\n\n> **Proxy Loss**: During training, this loss helps ensure that proxies distinctly and exclusively represent specific facial features. This distinctiveness is critical for selecting the appropriate experts for different facial characteristics.\n\nTo enforce that proxies represent mutually exclusive facial charachterstics, we initialize a fixed number (K) of equidistant and equiangular standard basis vectors. Each proxy is encouraged to align closely with its corresponding vector and diverge from the others. This is enforced through the following formulation:\n\n$$ L_{\\text{Proxy}} = \\frac{1}{K} \\sum_{i=1}^K \\left[ \\ln \\left( 1 + \\exp \\left( -\\alpha (s_{ii} - \\lambda) \\right) \\right) + \\frac{1}{|K-1|} \\sum_{k \\in K \\ \\text{and} \\ k \\neq i} \\ln \\left( 1 + \\exp \\left( \\beta (s_{ik} - \\lambda) \\right) \\right) \\right] $$\n\nIn this equation, $\\( s_{ik} \\)$ denotes the cosine similarity between the $\\(i^{th}\\)$ proxy and the $\\(k^{th}\\)$ basis vector. The first term of the equation encourages each proxy to approach its specific basis vector, while the second term ensures separation from the remaining vectors. Parameters $\\( \\alpha \\)$ and $\\( \\beta \\)$ scale the positive and negative similarities, respectively, with $\\( \\lambda \\)$ set at 0.1 as the threshold. This structured approach helps maintain the distinctiveness and efficacy of each proxy throughout training.\n\n2. Relying on feature-set distribution statistics may not capture fine-grained intra-set relationships as exhaustively as an O(n^2) approach such as attention. However, our method aims to approximate these relationships in O(n) through feature-set distribution statistics. We believe that our method accurately approximates these relationships based on our method’s superior performance to $O(n^2)$ methods such as CAFace [8] and CoNAN [5]. \n\n3. Thank you for your comment. Our method is specifically designed to excel in long-range and low-resolution data scenarios, addressing the challenges posed by such poor-quality inputs. Our method's superior performance in this poor-quality regime as evident from the performance on the Face Included Treatment setting of BTS3.1, where face images are uncontrolled and low quality.\n\n4. While we agree that this is an important aspect of biometrics and face recognition, our paper solely focuses on improving face feature fusion performance under challenging low-resolution and uncontrolled conditions and leaves adversarial robustness for future works. We would like to emphasize that adversarial study of face recognition methods is not the objective of this paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate the detailed and insightful feedback provided by the reviewers. We are thankful that the reviewers recognized the clear motivation and well-written presentation of our method (vG52, waA1), the extensive and rigorous experiments conducted, including empirical evidence on challenging datasets (waA1, BueQ), and the promising performance improvements over state-of-the-art approaches (vG52, waA1). We are also grateful for the acknowledgment of our method's linear time complexity and its invariance to feature order and set length (waA1, Cxak). Additionally, we appreciate the recognition of our paper's quality of writing (vG52, waA1) and the clarity and informativeness of our visualizations (waA1). 

Broadly the reviewers asked the following questions:
1. Reviewer vG52: how negative samples are obtained for the proxy loss and the strategy on handling several features per template
2. Reviewer waA1: code release, reproducibility and statistical significance
3. Reviewer BueQ: presentation of the method end experiment section and missing variable values
4. Reviewer Cxak: optimization and adversarial attacks

We have provided response to each question / point mentioned by the reviewers in detail below along with additional results as requested. We have further refined the text for typographical errors and readability as suggested by the reviewers. Furthermore, we have released our codebase on the following public anonymous github repository: https://github.com/anonymousdoubleblindreview/ProxyFusion

We thank the reviewers again for the constructive feedback that has helped improve the paper. We have been working diligently on addressing your critique and making necessary amends to the paper that will reflect in the final-version.

Meta-review: This submission proposed a face recognition framework for feature aggregation to deal with long-range, low-resolution cases. The reviewers recognized the merits of proposed method. Meanwhile, some concerns on presentation and writing are pointed.  3/4 reviewers are positive to this submission in general, while reviewer Cxak has some concerns on optimization details, potential drawback on feature extraction. After reading the submission, comments, and rebuttal, the AC agree with most of the reviewers and suggest accept this submission.
Decision: Accept (poster)