Learning Unified Representations for Multi-Resolution Face Recognition
Authors: Hulingxiao He
Wu Yuan
Yidian Huang
Shilong Zhao
Wen Yuan
HanQing Li
Venue: NeurIPS 2022 Submitted (2022)
Keywords: multi-resolution face recognition
deep representation learning

Abstract: In this work, we propose Branch-to-Trunk network (BTNet), a novel representation learning method for multi-resolution face recognition. It consists of a trunk network (TNet), namely a unified encoder, and multiple branch networks (BNets), namely resolution adapters. As per the input, a resolution-specific BNet is used and the output are implanted as feature maps in the feature pyramid of TNet, at a layer with the same resolution. The discriminability of tiny faces is significantly improved, as the interpolation error introduced by rescaling, especially up-sampling, is mitigated on the inputs. With branch distillation and backward-compatible training, BTNet transfers discriminative high-resolution information to multiple branches while guaranteeing representation compatibility. Our experiments demonstrate strong performance on face recognition benchmarks, both for multi-resolution identity matching and feature aggregation, with much less computation amount and parameter storage. We establish new state-of-the-art on the challenging QMUL-SurvFace 1: N face identification task.

As known, the resolutions of face images in reality may be far beyond the scope covered by the Learning Unified Representations for Face Recognition
Anonymous Author(s)
Affiliation
Address email
Abstract
In this work, we propose Branch-to- to-Trunk network (BTNet), a novel representation learning method for multi-resolution face recognition. consists of a trunk network (TNet), namely unified encoder, and multiple branch networks (BNets), namely resolution adapters. As per the input, resolution-specific BNet is used and the output are implanted as feature maps in the feature pyramid of TNet, at layer with the same resolution. The discriminability of tiny faces is significantly improved, as the interpolation error introduced by rescaling, especially up-sampling, is mitigated on the inputs. With branch distillation and boccward-coopatible training, BTNet transfers discriminative high-resolution information to multiple branches while guaranteeing representation compatibility. Our experiments demonstrate strong performance on face recognition benchmarks, both for multi-resolution identity matching and feature aggregation, with much less computation amount and param- eter storage. We establish new state-of-the-art on the challenging QMUL-SurvFacce 1: face identification task.
6
7
13
14
Introduction
15
Machine learning has advanced tremendously driven by deep learning methods, but is still severely challenged by various data specifications, such as data type, structure, scale and size, etc. For instance, face recognition (FR) is a well-established deep learning task, while the performance degrades dramatically in the testing domain that differs from the training one, influenced by factors of variance like resolution, illumination, occlusion, etc.
17
18
19
20
Most face recognition methods map each image to point embedding in the common metric space by deep neural networks (DNNs). The dissimilarity of images can be then calculated using various distance metrics (e.g., cosine similarity, Euclidean distance, etc.) for face recognition tasks.
23
24
Recent advancements in margin- loss (e.g., ArcFace [.. MV-Are-Softmax 2, CurricularFace 25 3 etc) enhanced discriminability of the metric space, with small intra-identity distance and large inter-identity distance. However, lack of variation training data still leads poor generalizability. Various useful methods are utilized to mitigate this issue. The model adapts to factors of variance 28 by augmenting datasets, whereas the large discrepancy in data distribution could potentially weaken
26
29 the model's ability to extract discriminative features with the same data scale and model structure
30 (see Section Fine-tuning is widely used to transfer large pretrained models to new domains with
31 different data specifications. However, this strategy requires one to store and deploy separate copy
32 of the backbone parameters for every single new domain, which is expensive and often infeasible.
34 model. As the small feature maps with fixed spatial extent (e.g.. 7 x 7) are mapped to an embedding
Submitted to 36th Conference Neural Information Processing Systems (NeurIPS 2022). Do distribute.

---

35
with a predefined dimension (e.g.. 128 - d, 512 d, etc.) by fully connected (fc) layer, input images need to be rescaled to a canonical spatial size (e.g., 112 x 1 112) before fed into the network. However, up-sampling low-resolution (LR) images introduces the interpolation error (see Section deteriorating the recognizable ones which contain enough clues to identify the subject. Even though super-resolution methods [4][0] are widely used build faces with good visualization, they inevitably introduce feature information of other identities when reconstructing high-resolution (HR) faces. This may lead to erroneous identity-specific features, which are detrimental to risk-controlled face recognition.
36
37
38
39
41
43
Empirically, we can divide inputs by resolution distribution and learn to operate on them via multiple models to achieve high accuracy and efficiency. However, directly for cross-resolution recognition as representation compatibility among models need to be guaranteed 15.
44
fashion cannot be applied
47
To improve discriminability while ensure the compatibility of the metric space for multi-resolution face representation, we learn the "unified" representation by partially-coupled Branch-to- to-Trunk Network (BTNet). It is composed of multiple independent branch networks (BNets) and a shared 50 trunk network (TNet). A resolution-specins BNet used for a given image, and the output are implanted feature maps in the feature pyramid of TNet, at layer with the same resolution.
48
49
51
52
Furthermore, we find that multi-resolution training can be beneficial to building strong and robust TNet, and backward-compatinn training (BCT) [1] can improve the representation compatibility during the training process of BTNeet. To ameliorate the discriminability of tiny faces, we propose branch distillation intermediate layers, utilizing information extracted from HR images to help the extraction of discriminative features for resolution-specifie branches.
53
54
55
56
Our method is simple and efficient, which breaks the convention of up-sampling the inputs and serves as a general framework that can be easily implemented by several existing methods due to conceptual simplicity. Meanwhile, BTNet is able to reduce the number of FLOPS by operating the inputs without up-sampling. and per-resolution storage cost by only storing the learned branches and resolution-aware BNs [16], while re-using the copy of the trunk model.
58
59
60
61
We demonstrate that our method performs comparably in various open-set face recognition tasks (1:1 face verification and 1: N face identification), in both settings of multi-resolution identity matching and feature aggregation, while meaningfully reduces the redundant computation cost and parameter storage. In the challenging QMUL-SurvFad 1: N face identification task [17] we establish new 65
63
64
66 state-of-the-ane by outperforming prior models. Furthermore, by avoiding the ill-posed problem (i.e.,
87
image up- sampling),) our approach also effectively reduces the additional noise and uncertainty of the representation, which plays key role in reliable risk-controlled face recognition.
2
Related Work
69
70
Compatible Representation Learning: The task of compatible representation learning aims at
71
encoding features that are interoperable with the features extracted from other models. Shen et. al. [1 first formulated the problem of backward-compatibl learning (BCT) and proposed to utilize the
73 old classifier for compatible feature learning. Since the multi-model fashion benefits representation
74 learning with lower computation, our idea of cross-resolution representation learning can be modeled
75 similar to cross-model compatibility [1H15), as metric space alignment for different resolutions. Our goal is achieved by both compatibility-ave aware aware network architecture and training strategy.
76
77
Knowledge Distillation and Transfer: The concept of knowledge distillation (KD) was first proposed by Hinton et. al. in [18) which can be summarized as employing a large parameter model (teacher) supervise the learning of small parameter model (student). Distillation from intermediate features 29 is widely adopted to enhance the effectiveness of knowledge transfer. However, due to the "dark knowledge" hidden in the intermediate layers, additional subtle design is often required to match and rescale intermediate features. Instead, our approach can easily locate the
78
79
81
82
83 distillation features without rescaling and effectively transfer knowledge from the HR domain to LR
84 branches.

---

85
Low Resolution Face Recognition: Its task includes low resolution-to-low resolution (LR-to-LR) matching and low resolution-to-high resolution (IR-tt-HRR matching [30]. The work can be divided into two categories [1: (1) Super-resolution (SR) based methods aim to upscale LR images to construct HR images and use them for feature extraction [4 (2) Projection-based methods aim to extract adequate representations different domains and project them into a common feature space SR approaches are able build faces with good visualization, but inevitably introduce feature information of other identities when reconstructing corresponding HR faces, thus introducing noise for identity-specife features. Compared to previous projection methods, our approach directly learns discriminative representations in a common feature space for HR and LR inputs, without additional projection heads for feature transformation.
86
87
86
89
32 34
91
93
94
95
Pseudo-Sian -eeume- Networks: Siamese networks are coupling architecture based DNNs, which widely used for signature verification [35), face verification (3), tracking [8), etc. Pseudo- Siamese networks [39] are decoupled Siamese networks, as the weights of the two branches are not shared, resulting in a more flexible representation way for the two entities. Hughes et. al. in [0] proposed a pseudo-Siamese CNN for identifying corresponding patches in SAR and optical images. Inspired by pscudo-Siamese networks, we propose a resolution-adaptii partially coupled Siamese network architecture, extracting specific-shared features for images with different resolutions.
97
98
89
100
101
3
Learning Specific-Shared Feature Transfer
102
103
Instead of rescaling the inputs to canomical size, we build multiple resolution-specdt branches (BNets) that are used to map inputs to intermediate features with the same resolution and a resolution-
104
105
shared trunk (TNet) to map feature maps with different resolutions to a sigh-dimension embedding. gain several important properties by doing so: (1) Processing inputs on its original resolution can diminish the inevitably introduced error via up-sampling or information loss via down-sampling, thus preserving the discriminability of visual information with different resolutions. (2) Information streams of different resolutions are encoded uniformly, thus enabling the representation compatibility, which is particularly beneficial to open-set face recognition considering that compatible metric space is the prerequisite for computing similarity. (3) This also effectively reduce the computation
106
107
110
111
112
for LR images by supplying computational resources conditioned on the input resolution.
113
3.1
Up-Sampling Error Analysis
Encode
20.0
17.5
15.0
TNet
EStiated entor
12.5
10.0
Adapt
BNets
Images difeent reedulese iienity
AAAA
Imagen with difiees resdutioos identty
100
Image resslution
Figure 2: Basic ideas of the proposed BTNet.
Figure 1: Estimated Error Upperbound. (bilinear interpolation, average value for over 100 images) with the change of image reso- lution relative to resolution 12.
Images of a certain identity are first projected to the feature maps with the same resolution respectively (Adapt) and then projected to unified feature repre- sentation (Encode). In this figure, feature maps with the same resolution are indicated by outlines in the same color.
114 Figure illustrates the experimental estimation interpolation error, whose upper bound increases
115 with the decline of the image resolution (see detailed theoretical derivation in Appendix A. Note that the error soars up when the resolution drops below 32 approximately which can be viewed as LR face images, consistent with the tiny-object criterion 41

---

118
The results show that: (1) inputs with resolution higher than around 32 can be considered in the same HR domain, since the error information introduced by up-sampling via interpolation can be ignored to a certain extent; (2) inputs with a resolution lower than around 32 should be treated as in various LR domains due to the high sensitivity of the resolution to errors.
119
120
121
122
3.2
Branch-to- tranch-to-Trunk Network
Let X be an input RGB image with a space shape: X E "Rxx where H x W corresponds to the spatial dimension of the input. For efficient batch training and inference, we predefine a canonical
123
124
125
size (e.g.. 112 112 for typical face recognition models like ArcFace I..
We build trunk network T H x3
Camb capable of extracting discriminative information with different resolutions, where Cemb is the number of embedding channels. For every resolution in the candidate set, we formulate a resolution-specin branch, 2, = B, (X,), which maps the imput image X, to feature maps with the same resolution and expanded channels 2, RRxxxx3 - RrxxxCr
128
129
130
The idea is to learn our branches B to focus on resolution-specinee feature transfer independently. Feature maps will then be coupled to the trunk network in the feature pyramid with the same spatial resolution r r. allowing for further mapping the unified presentation space by T, RrxxxC R CCmbs
131
132
134
Here, we follow the idea of "avoiding redundant up-sampling". Our branches B are implemented with same-resolution mapping: i.e., the model preserves the network architecture of from input to the layer with resolution r and abandons lown-sampling operations (e.g., replacing the convolution of stride with stride 1, abandoning the pooling layers, etc.) to keep the resolueso flow.
135
136
137
We specifically name specific-shared feature transfer network as Branch-to- to-Trunk Network, abbreviated as "BTNet". Figure 2 visually summarizes the main ideas of BTNet.
139
140
3.3
Training Objectives
We now describe the training objectives. The training of BTNet includes training the trunk network T such that it can produce discriminative and compatible representations for multi-resolution infor- mation, and fine-tuning the branch networks B to encourage them to learn resolution-specife feature transfer, so as to improve accuracy without compromising compatibility.
148
144
145
Influence Loss. It is a compatibility-aww aware classification loss which is implemented by feeding the embeddings of the new model the classifier of the old model [T]. Since the difficulties of samples vary due to image resolution, we compute CurricularFace as our classification loss, in the form of:
146
147
148
E cootO +m)
Lour - log( e ccos(0, +m)
(I)
N(t, cos (os( (t) cos(O,)),
cos(On, + m) - cos((,)) 2
(2)
else
(1)) = Σ cos 0y, + (1 - a) ((6-1)
150 which distinguishes both the difficultness of different samples in each stage and relative importance of easy and hard samples during different training stages. Thus, refine CurrieularFace loss as influence loss:
Lin Linfluence =
Leaur Leaur(pbbe k*)
(4)
153 where sob is BTNet backbone (both B, and T,), and n² is the classifier of the pretrained trunk T.

---

154
Branch Distillation Loss. Due to the
155
continuity of the scale change of both the
156
image pyramid and the feature pyramid
157
42), we can get a qualitative sense the similarity between images and feature maps with the same resolution (see Figure Furthermore, features extracted from HR images have richer and clearer infor- mation than those from LR images [43]. Motivated by these analyses, we utilize an MSE loss to encourage the branch output
158
3
182
56x56
28x28
14x14
163
Figure 3: Visual comparison of face image-feature map pairs with different resolutions (resized to common size here for illustration).
164
2, be similar the corresponding fea- 165
168
ture maps of the pretrained trunk network
Zgi
Lbranch
Σ (2r. Zau
(5)
168
where V denotes the batch size.
The whole training objective is combination of the above objectives:
L Lin fluence + Abranch Lirrnnh
(6)
170
where Abranch is a hhper-paramette to weigh the losses and we set Abranch = 0.5 in all our experi-
171 ments.
Fally Fontaring
Rox Par
10.
213
412
Figure 4: Comparison of # Params (M) be- tween fully finetuning and por-
Figure 5: Comparison of FLOPs (G) between baselines and ppt-
3.4 Storing Branch Networks
173
An obvious adaptation strategy is fully finetuning of the model each resolution. However, this strategy requires one to store and deploy separate copy of the backbone parameters for every resolution, which is an expensive proposition and difficult to expand into more segmented resolution branches. Our BTNet beneficial in the scenario of multi-resolution face recognition which achieves better parameter/accuraacy trade-offs. Since activation statistics including means and variances under different resolutions are incompatible [44], we update and store Batch Normalization (BN) [45] parameters in all layers of B, and T, for each resolution, whose amount negligible. Apart from this, only need store the learned branches and re-use the original copy of the pretrained trunk model, significantly reducing the storage cost. Figure shows that BTNet requires only 1.1% 48.9% of all the parameters compared to fully updating all the parameters of TNet.
175
178
177
178
179
180
182
Experiments
To validate BTNet face recognition tasks in open universe, perform 1:1 verification and 1 N identification tasks in two different settings, including (a) multi-resolution identity matching, and
5

---

186
(b) multi-resolution feature aggregation. For 1:1 verification, a pair of templates are provided and the model is to decide whether they belong to the same identity or not. For 1:N identification, a set of gallery images are first mapped onto their embedding vectors (indexing) and the embeddings of query images are extracted to perform search against indexed gallery.
187
188
189
190
4.1
Implementation Details
We use MSIMv3 [46] for training face embedding models. The MS1Mv3 dataset contains 5,179,510 images of 93,431 celebrities. According to the test setting, different test datasets
Datasets.
192
193
are used.
194
-Mult-Ressolution Identity Matching. We try on six widely adopted face verification benchmarks: LFW [47], CFP-FF [48]. CFP-FP (48), AgeDB 30 49,, CALFW [50]. and CPLFW [5T]. while the large-scale surveillance face dataset QMUL-Surv 17 is used for 1:N face identification,
195
196
197
which contains native LR surveillance faces across wide space and time. The spatial resolution for
OMUL-SurVaace ranges from 6/5 to 124/106 in height/width with average of 24/20.
Multi- -estiiReesolttonn Feature Aggregation. We adopt top challenging benchmark IJB-C C [52]. which has around 130k images from 3,531 identities, for two standard testing protocols: 1 verification and I:N identification.
200
202
Training. All the models are trained on four RTX 2080 Tis with batch size 128 by stochastic
203
gradient descent. For TNett, we train for 25 epochs, with learning rate initialized at 0.2 with 2 warm-
204
up epochs and decaying as a quadratic polynomial. We augment training samples by random horizonal flipping and multi-resolution training. For BNets, we initialize the learning rate by 0.02 without warm-up epochs. The training all stops at the 10th epoch for fair comparison. The recommended used for classification loss from the original paper (e.g., 0.5, 64 for ArcFace 0, and 0.99, 1 0 for CurricularFace Only horizonal flipping is used as augmentation when training BNets.
205
hyper-parametery
209
210
In our experiment, several baselines are used to validate BTNet in learning discriminative and compatible representations for multi-resolution face recognition.
Baselines.
211
212
High- Trained Plu- Naive baseline trained with HR data.
213
Independently Trained fmm- Multi-modt model fashion: is it possible to achieve better results if we train a specific model for each resolution independently? Specifically, we train yrr for data with resolution r and denote the multi-model collections as Pmmm
214
215
216
Trained Pmr- Trained with multi-resolution data which adapts resolution- variance. Specifically, each image randomly down-sampled to size in the candidate set ( 1122 x 0, 1, 3, 4] with equal probability of being chosen, and then up-sampled back to 112 112.
112
2
Instantiation of Network Architecture. The BTNet and baselines are implemented with ResNer50 [3 and they could be extended easily with other implementations. Dubbed as Pae, the detailed instantiation of BTNet based on ResNet50 is illustrated in Appendix A.2.
220
221
222
4.2
Evaluation Metrics
223
On the benchmarks for face verification, we use 1:1 verification accuracy the basic metries. The rank- 20 true positive identification rates (TPIR20) at varying false positive identification rates (FPIR) and AUC are used to report the identification results on QMUL-SurvFace, The evaluation metrics for UJBC 1:1 verification protocol are true acceptance rates (TAR) at different false acceptance rate (FAR). For I:N identification, the basic evaluation metrics are the true positive identification rates (TPIR) at different false positive identification rates (FPIR).
226
227
228
229
For better evaluation, we define another two metrics to assess the relative performance gain similar
14

---

Table 1: Comparison of different methods on six face verification benchmarks. "Acc." denotes average 1:1 verification accuracy.
(a) Cross-resolution identity matching.
(b) Same-resolution identity matching.
11287
1128214
112422
14814
288228
1128112
Acc:
Cain
Aoc.
Gain
Acc.
Acc.
Gain
Phr
57.75
81,02
95.90
73,88
95.58
97.68
Pmmm
50.58
-0.89
49.50
-482
50.03
305.80
62.57
+1.00
78.00
+1.00
94,68
+1.00
97.68
65.85
+1.00
87.47
+1.00
96.05
+1.00
61.02
+0.17
80.32
+1.56
95.12
+1.40
97.25
(Chrrs)
$6.10
+3,50
94.08
42,02
96.65
+5.00
77.78
+9.13
90.90
+4.13
96.27
+2.45
97.25
Cross-Resolution Gain.
With the purpose towards the cross-resolution compatible representations,
we define the performance gain as follows:
&cr Mr, (p) My, &er., (pPhr)
[Mr (Fmm) Mrr&er (4Phr)
Here My (+) are metrics when the resolutions of the image/template pair are ri X TI and 72 72 (r1 # r2), respectively. Pmr shares the same architecture with Por while is trained on multi nulti-resolution images and thus serves as the baseline of cross-resolution gain.
234
235
Gain. For the scenario of multi-resolution face recognition, the performance of same-resolution verification//deetfficaaio is also vital besides cross-resolution one. Therefore, we report the relative performance improvement from base model fhr in the scenario of same-resolution.
237
238
(y) Mr (pprr
Gain (gp) |Mr&r (5pr) Mr&rr(Phrr
(8)
Here Mr&r (.) are metrics when the resolutions of the image/template pair are both r X r. (Pr is model of the set {1mm = /rrrr 7, 14, 28} trained on images with resolution considering cross-resolution representation compatibility, which serves as the baseline of same- resolution gain on resolution r. Note that for both metrics we add the absolute symbol to the denominator as they can be negative in some test settings (detailed in Section and and
without
241
242
243
244
4.3
Multi-Resolltion Identity Matching
We now conduct experiments on the proposed BTNet framework for multi-resolution identity match- ing. Two different settings are included (1) same-resolution matching, and (2) cross-resolution matching. Table I
247
compares the average performance on popular benchmarks for Phrr Pmmm Pmr The experimental results on each dataset are detailed in Appendix A.5.
248
Pbx.
249
When directly applied to test data with the resolution lower than training data, fhr suffers a severe performance degradation. Up-sampling images via interpolation can increase the amount of data but not the amount of information, only to improve the detailed part of the image and the spatial resolution (size) (44) Moreover, it also brings various noise and artificial processing traces (65] Up-sampling images via pnerpolation-ttpicallll bilinear interpolation or bicubic interpolation of 4x4 pixel neighborhoods, essentially function approximation method, is bound to introduce error information (detailed Appendix 1) thus potentially confusing identity information, which is especially crucial for LR images with limited details. We are able to observe improvement of Pm in same-resolution matching but its cross-resolution gain negative with approximately 50% accuracy. Unsurprisingly, independently trained y is unaware of representation compatibility, and thus does not naturally suitable for cross-resolution recognition. The results show that yor improved both cross-resolution and samc-resolution accuracy by large margin, as it learns to adapt to resolution variance and maintain discriminability of melti-resolutionn inputs. Note that the model size and training data scale stay the same, while only the resolution distribution of the data changes for Pmr, and thus there is a marginal accuracy drop in the setting of 112& 12 matching. Comparably, Pbr substantially outperforms all baselines with 2.02 ~5.00 cross-resolution gain and 2.45-9.13 same-resolution gain. Importantly, due the multi-resolutinn branches, our approach has cost same
250
253
254
255
256
257
280
261
262
263
264
with fro. significantly lower than (hh and pmr (see Figure 5)
7

---

Table 2: Performance of face identification on QMUL-SuryFace. Most compared results are cited
from [7 54 except BTNet.
TRIR3XSJ0FPIR
AUC
VGG-Face 55
2.6
DeepID2
50
20.8
128
8.1
3.4
FaceNet S
19.8
127
8.1
4.3
SphereFace
62
28.1
21.3
15.7
8.3
1.0
SRCNN 30
27.0
20.0
14.9
6.2
0.6
ESRCNN 60
27.3
20.0
14.4
6.1
0.7
VDSR 61
27.3
20.1
14.5
6.1
0.8
DRRN 02
27.5
20.3
14.9
6.3
0.6
LapSKN
27.4
20.2
14.7
6.3
0.7
ArcFace
25.3
18.7
15.1
10.1
2.0
RAN 5
32.3
26.5
21.6
14.9
BT (acg.efloo))
32.6
27.9
23.4
16.5
NeN (ang.+aeup
34.6
20.3
25.7
18.9
Nel (ong-+oeil)
35.4
31.1
26.8
20.3
BTN (min--loor)
32.3
27.6
23.2
(min--neuz)
(max=+lone)
RTN
31.0
26.4
BTNet (maxeceil)
35.4
31.2
287
For inference on inputs with resolutions not strictly matched to the branch, we validate three selection strategies based on three resolution indicators (see Figure 6). Table compares BTNet against the state-of-the-ane models on QMUL-SurvFace 1:N identification benchmark. We are able to observe that our proposed approach extends the state-of-the -hetthe-rrss while being more computationally efficient. We believe the performance of BTNet (max ceil) is the highest that have been reported far, and believe it meaningful with the increased focus unconstrained surveillance applications.
288
289
270
273
4.4 Multi-Resolution Feature Aggregation
Multi-resolution feature aggregation is common in set-based recognition tasks where the model needs to determine the similarity of sets (templates), instead of images. Each set could contain images of the same identity with different resolutions. In our experiment, we rescale the original and flipped images in each set to different resolutions and aggregate their features into a representation of the template. Detailed experimental results can be seen in Appendix A.5.
278
277
278
Table compares the cross-resolution results of TAR @FAR=10 for 1:1 verification. The resolutono features ensured to be mapped the same vector space where the aggregation conducted for yPhr and pmr. but possible reason is that Phr has outstanding discriminability to extract HR features, while LR features may not overly deteriorate the HR information. This phenomenon also suggests that Pmr sacrifices its discriminability in exchange for the adaptability for resolution-variance We can see Po. is comparable with Phr, demonstrating the discriminative power BTNet for aggregating multi-resolution features.
observe that Phr performs much better than Pmr. One
282
283
284
285
Table (b) compares the same-rcsolution results of TAR @FAR=10- for 1:1 verification. When HR information is removed from the template representation (i.e., test settings 7&7, 14&:14, 28&:28),
287
288
Phr suffers from performance degradation as well, as the informative embedding cannot catch the (54] Both Pmm and (mr improve with a limited same-resolution gain, while Por surpasses the baselines by large margin while also reducing the compute.
289
lost details of the LR images
290
In Table we show the results of TPIR @FPIR=10-1 for 1:N identification protocol. Similar to results for 1:1 verification, are able to observe that Pbu is comparable or even better than Par with

---

Table Comparison of different methods on the IJB-C dataset 1:1 face verification task. "TAR" denotes TAR (%@FAR (e@FAR=-))))
(a) Cross-resolution feature aggregation.
(b) Same-resolution feature aggregation.
1124228
70.7
14814
TAR
Gain
TAR
Gain
Gain
PAr
$8.89
92.40
95.62
4.53
89.65
96.40
Pomm
74.54
-0.56
93.52
+1.33
95.42
-0.09
4.83
0.00
29.26
-1.00
92.58
+1.00
96.40
Por
63.11
-1.00
91.56
-1.00
95.33
-1.00
4.48
40.51
+1.51
92.81
+1.08
96,05
(Chrss)
88.17
-0.03
93.97
+1887
95.62
+0.00
35.47
82.08
+10.79
94.50
+1.66
96.06
Table 4: Comparison of different methods on the IJB-C dataset 1: N face identification task. "TPIR" denotes TPIR (% @ FFIR=O.,),
(a) Cross-resolution feature aggregation.
(h) Same-resolution feature aggregation.
11287
112814
1126228
767
14&14
28828
1126112
TPIR
Gain
TPIR
Gain
TPIR
Guin
TPIR
Gain
TPIR
Gain
TPIR
Gain
94.27
26.37
Ponm
-0.33
+1.00
2184
-1.00
+1.00
95.57
Pour
$6.64
89.05
-1.00
37.58
+2.47
+1.34
(hr(Ours)
83.03
94.33
+204,33
76.65
+11.10
+1.85
293
HR information involved and can preserve superior discriminability with limited LR information, while also being more computationally efficient.
294
W
max
floor
W
Fim
H
min
near
10
14
Tout E (7,14,28,112))
H
avg
ceil
B
B
Figure 6: Branch selection process. Max/min/avverag used on (W, H) to obtain a resolution indicator for further allocation (florr/nearccii) certain branch.
5
Discussion and Conclusion
285
296
This paper works on the problem of multi-resolution face recognition, and provides a new scheme to operate images conditioned on its input resolution without large span rescaling. The error intro- duced by up-sampling via interpolation is investigated and analyzed. Decoupled as branches for discriminative representation learning and coupled as the trunk for compatible representation learning, multi-resolution face
297
298
289
Branch-to- -oruukkk Network (BTNet) achieves significant improvements verification and identification tasks. Besides, the superiority of BTNet in reducing computational cost and parameter storage cost is also demonstrated. It is worth noting that our approach is easy to expand to recognition tasks for other classes of objects and has the potential to serve as a general network architecture for multi-resolution visual recognition.
302
303
304
Limitations and Future Work. The dislocation between the underlying optical resolution of native face images and that of a certain branch may limit the power of the model, which may be improved by selecting the optimal processing branch for the input in combination with the image quality, rather than by image size alone. The optimal branch selection strategy is not fully investigated though we
306
307
308
309
have provided an intuitive way to select the branch for inputs (see Figure 6). Importantly, based on
310
the unified multi-resolltion metric space, the underlying resolution of the imputs (integrated spatial
resolution with quality assessment) can be utilized provide the reliability of the representation and
contribute risk-controlled face recognition. They will be our future research directions.

---

References
313
314
[1] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. In IEEE Conference on Computer Vision and Puttern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019. pages 4690-4699, Computer Vision Foundation IEEE, 2019.
315
316
317
318
[2]
Xiaobo Wang, Shifeng Zhang, Shuo Wang, Tianyu Fu, Hailin Shi, and Tao Mei. Mis-classified vector guided softmax loss for face recognition. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Secone Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 12241- 12248. AAAI Press, 2020.
319
320
322
324
[3] Yuge Huang, Yuhan Wang, Ying Tai, Xiaoming Liu, Pengcheng Shen, Shaoxin Li, Jilin Li, and Feiyue Huang. Curricularface: Adaptive curriculum learning loss for deep face recognition. In 2020 IEEF/CVE Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 5900-5909, Computer Vision Foundation IEEE, 2020.
326
327
[4] Shizhan Zhu, Sifei Liu, Chen Change Loy, and Xiaoou Tang. Deep cascaded bi-network for face hallucination. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision - ECCV 2016, pages 614 630, Cham, 2016. Springer International Publishing.
329
330
[5] Klemen Grm, Walter J. Scheirer, and Vitomir Struc. Face hallucination using cascaded super- resolution and identity priors. IEEE Trans. Image Process., 19:2150-2 2020.
338
[6]
Zhangyang Wang. Shiyu Chang, Yingzhen Yang. Ding Liu, and Thomas S. Huang. Studying very low resolution recognition using deep networks. CoRR, abs/1601.0413. 2016.
335
Zhiyi Cheng, Xiatian Zhu, and Shaogang Gong. Low-resolution face recognition. In C. V. Jawahar, Hongdong Li, Greg Mori, and Konrad Schindler, editors, Computer Vision ACCV Computer Vision, Perth, Australia, December 2-6, 2018,
336
2018- 14th Asian Conference Revised Selected Papers, Part III, volume 1363 of Lecture Notes in Computer Science, pages 605-621. Springer, 2018
338
339
[8] Xi Yin, Ying Tai, Yuge Huang, and Xiaoming Liu. FAN: feature adaptation network for surveillance face recognition and normalization. In Hiroshi Ishikawa, Cheng-Lin Liu, Tomás Pajdla, and Jianbo Shi, editors, Computer Vision ACCV 2020 15th Asian Conference on Computer Vision, Kyoto, Japan, November 30 - DDeembee 4, 2020, Revised Selected Papers, Purt II, volume 12623 of Lecture Notes in Computer Science, pages 301-319. Springer, 2020.
341
342
343
344
345
[9]
Maneet Singh, Shruti Nagpal, Richa Singh, and Mayank Vatsa. Dual directed capsule network for very low resolution image recognition. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2, 2019, pages 340-349, IEEE, 2019.
346
347
348
349
[10]
Aashish Rai, Vishal M. Chudasama, Kishor P. Upla, Kiran B. Raja, Raghavendra Ramachandra, and Christoph Busch. Comsupresnet: A compact super-resolution network for tow-resolution face images. In 8th International Workshop on Biometrics and Forensics, IWBF 2020, Porto, Portugal, April 29-30, 2020, pages 1-6. IEEF, 2020.
350
351
352
353
[1I] Yantao Shen, Yuanjun Xiong, Wei Xia, and Stefano Soatto. Towards backward-compatible representation learning. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 6367-6376. Computer Vision Foundation IEEE, 2020.
354
357
[12] Mateusz Budnik and Yannis Avrithis. Asymmetric metrie learning for knowledge transfer. In IEEE Conference Computer Vision and Pattern Recognition, CVPR 2021, virtual. June 19-25. 2021, pages Computer Vision Foundation/ / IEEE, 1021.
10

---

380
[13] Chien-Y Yi Wang, Ya-Liang Chang, Shang- Ta Yang, Dong Chen, and Shang- Hong Lai. Unified representation learning for cross model compatibility. In 31st British Machine Vision Conference 2020, BMVC 2020, Virtual Event, UK, September 7-10, 2020. BMVA Press, 2020.
361
362
363
[14] Qiang Meng, Chixiang Zhang, Xiaoqiang Xu, and Feng Zhou, Learning compatible embeddings. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC. Canada, October 10-17, 2021, pages 9919-9928. IEEE, 2021.
384
385
[15] Rahul Duggal, Hao Zhou, Shuo Yang, Yuanjun Xiong, Wei Xia, Zhuowen Tu, and Stefano Soatto. Compatibility-awann heterogeneous visual search. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, viriual, June 19-25, 2021, pages 10723-10732. Computer Vision Foundation / IEEE, 2021.
389
370
[16] Mingjian Zhu, Kai Han, Enhua Wu, Qiulin Zhang, Ying Nie, Zhenzhong Lan, and Yunhe Wang. Dynamic resolution network. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6- 14, 2021, virtual, pages 27319-27330, 2021.
371
372
373
374
[17] Zhiyi Cheng. Xiatian Zhu, and Shaogang Gong. Surveillance face recognition challenge. CoRR, abs/1 has/1804.0999) 2018.
376
377
[18] Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. CoRR, Ab0.0550,,0,,,, 2015.
378
379
[19] Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi. A comprehensive overhaul of feature distillation. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 -Nvember 2, 2019, pages
380
381
382
1921-1930. IEEE, 2019.
383
[00] Zehao Huang and Naiyan Wang. Like what you like: Knowledge distill via neuron selectivity transfer. CoRR, abs/1 7bb//177,0,119,, 2017.
[21] Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA. USA, June 16-20, 2019, pages 3967-3976. Computer Vision Foundation / IEEE, 2019.
387
388
[22] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. 2015.
389
390
391
392
[23] Frederick Tung and Greg Mori. Similarity knowledge distillation. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2, 2019, pages 1365-1374. IEEE, 2019.
[24] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Richard C. Wilson, Edwin R. Hancock, and William A. P. Smith, editors, Proceedings of the British Machine Vision Conference 2016, BMVC 2016, York, UK, September 19-22, 2016. BMVA Press, 2016.
396
397
398
[25]
Junho Yim, Donggyu Joo, Ji-Hoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In 2017 IEFE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 7130-7138. IEEE Computer Society, 2017.
389
400
401
[26] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. CoRR,
abs/1910.10699, 2019.

---

404
[27] Baoyun Peng, Xiao Jin, Dongsheng Li, Shunfeng Zhou, Yichao Wu, Jiaheng Liu, Zhaoning 7.aagg and Yu Liu. Correlation congruence for knowledge distillation. In 2019 IEEF/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2, 2019, pages 5006-5015, IEEE, 2019.
405
406
407
408
[28] Jangho Kim, Seonguk Park, and Nojun Kwak. Paraphrasing complex network: Network compression via factor transfer. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Sesa-Bianchi, and Roman Garnett, editors, Advances in Neural Information
409
410
411
Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pages 2765-2774, 2018.
412
413
[29]
Byeongho Heo, Minsik Lee, Sangdoo Yun, and Jin Young Choi. Knowledge transfer via distillation of activation boundaries formed by hidden neurons. In The Thirty-Thine AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 February 1, 2019, pages 3779-3787. AAAI Press, 2019.
414
415
416
419
[30]
Yoanna Martínez-Díaz, Heydi Méndez- Vázquez, Luis S. Luevano, Leonardo Chang, and Miguel González-Mendozz, Lightweight low-resolution face recognition for surveillance applications. In 25th International Conference on Pattern Recognition, ICPR 2020, Virtual Event / Milan, Italy, January 10-15, 2021, pages 5421-5428 IEEE, 2020.
420
423
[31] Luis S. Luevano, Leonardo Chang, Heydi Méndez- Méndez-Vázquez, Yoanna Martínez-Díaz, and Miguel González-Mendozze study on the performance of unconstrained very low resolution face recognition: Analyzing current trends and new research directions. JEEE Access, 9:75470- 75493, 2021.
424
425
427
[32] Ze Lu, Xudong Jiang, and Alex C. Kot. Deep coupled resnet for low-resolution face recognition. IEFE Signal Process. Lett., 25(4):526-530, 2018.
428
429
[33] Sivaram Prasad Mudunuri, Soubhik Sanyal, and Soma Biswas. Genlr-net: Deep framework for very low resolution face and object recognition with generalization to unseen categories. In 2018 IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 489.498. Computer Vision Foundation IEBE Computer Society, 2018
430
431
432
433
434
[34] Juan Zha and Hongyang Chao. TCN: transferable coupled network for cross-resolution face recognition In IEEE International Conference Acoustics. Speech and Signal Processing, ICASSP 2019, Brighton, United Kingdom, May 1 12-17, 17, 2019, pages 3302-3306. IEEF, 2019.
437
[35] Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. Signature verification using a siamese time delay neural network. In Jack D. Cowan, Gerald Tesauro, and Joshua Alspector, editors, Advances in Neural Information Processing Systems 6, [7th NIPS Conference, Denver, Colorado, USA, 1993], pages 737-744. Morgan Kaufmann, 1993.
438
440
441
[36]
Yaniv Taigman, Ming Yang, Mare' Aurelio Ranzato, and Lior Wolf. Deepface: Closing the gap to human-level performance in face verification. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2014, Columbus, OH, USA, June 23-28, 2014, pages
442
1701-1708. IEEE Computer Society, 2014.
445
[37] Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning a similarity metric discriminatively, with application to face verification. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2005), 20-26 June 2005, San Diego. CA, USA, pages 539-546. IEBE Computer Society, 2005.
446
12

---

449
[38] Luca Bertinetto, Jack Valmadre, João F. Henriques, Andrea Vedaldi, and Philip H. S. Torr. Pully-convolutional siamese networks for object tracking. In Gang Hua and Hervé Jégou, editors, Computer Vision - ECCV 2016 Workshops Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part II, volume 9914 of Lecture Notes in Computer Science, pages 850-865, 2016.
450
451
452
453
[39] Sergey Zagoruyko and Nikos Komodakis. Learning compare image patches via convolutional neural networks. In IEEE Conference Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pages 4353-4361. IEEE Computer Society, 2015.
457
[40] Lloyd H. Hughes, Michael Schmitt, Lichao Mou, Yuanyuan Wang, and Xiao Xiang Zhu. Identifying corresponding patches in SAR and optical images with a pseudo-siamese CNN. IEEE Geosci. Remote. Sens. Lett., 15(5):784-788, 2018.
458
459
460
[41]
Antonio Torralba, Robert Fergus, and William T. Freeman. 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE Trans. Puttern Anal. Mach. Intell., (111)):958-19700 2008.
462
[42] Tony Lindeberg. Scale-space theory: a basic tool for analyzing structures at different scales. Journal of Applied Statistics, 21(1-2):225- 2200 1994.
464
485
[43] Yui Man Lui, David Bolme, Bruce A. Draper, J. Ross Beveridge, Geoff Givens, and P. Jonathon Phillips. A meta-analysis face recogmition covariates. In 2009 IEEE 3rd International Conference on Biometrics: Theory, Applications, and Systems, pages 1-8, 2009.
486
467
468
[44] Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Hervé Jégou. Fixing the train-test resolution discrepancy. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8 14, 2019, Vancouver, BC, Canada, pages 8250-820066 2019.
489
471
472
473
[45]
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Francis R. Bach and David M. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pages 448-456. JMLR.org. 2015.
474
475
476
[46] Jiankang Deng, Jia Guo, Debing Zhang, Yafeng Deng, Xiangju Lu, and Song Shi. Lightweight face recognition challenge. In 2019 IEEE/CVF International Conference on Computer Vision Workshops, ICCV Workshops 2019, Seoul, Korea (South), October 27-28, 2019, pages 2638- 2646. IEEE, 2019.
479
480
481
482
[47] Gary B Huang, Marwan Mattar, Tamara Berg, and Eric Learned-Mille Labeled faces in the wild: A database forstudying face recognition in unconstrained environments. In Workshop on faces in' Real-Life Images: detection, alignment, and recognition, 2008
483
484
[48] Soumyadip Sengupta, Jun Mun-Cheng Chen, Carlos Castillo, Vishal M Patel, Rama Chellappa, and David W Jacobs. Frontal to profile face verification in the wild. In 20/6 IEEE winter conference on applications of computer vision (WACV), pages 1-9, IEEE, 2016.
486
487
488
[49] Stylianos Moschoglou, Athanasios Papaioannou, Christos Sagonas, Jiankang Deng, Irene Kotsia, and Stefanos Zafciriou. Agedb: The first manually collected, in-the-wild age database. In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2017, Honolulu, III, USA, July 21-26, 2017, pages 1997-2005. IEEE Computer Society, 2017.
489
490
481
[50] Tianyue Zheng, Weihong Deng, and Jiani Hu. Cross-age LFW: A database for studying cross-age face recognition in unconstrained environments. CoRR, abs/ 170.008887. 2017.
13

---

494
[51] Tianyue Zheng and Weihong Deng. Cross-poct pose Ifw: A database for studying cross poss--pose face recognition in unconstrained environments. Beijing University of Posts and Telecommunications, Tech. Rep, 5:7, 2018.
485
496
487
[52]
Brianna Maze, Jocelyn C. Adams, James A. Duncan, Nathan D. Kalka, Tim Miller, Charles Otto, Anil K. Jain, W. Tyler Niggel, Janet Anderson, Jordan Cheney, and Patrick Grother. IARPA janus benchmark- C: face dataset and protocol. In 2018 International Conference on Biometrics, ICB 2018, Gold Coast, Australia, February 20-23, 2018, pages 158-165. IEEE, 2018
498
499
500
501
[53] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778. IEEE Computer Society, 2016.
502
504
[54] Han Fang. Weihong Deng, Yaoyao Zhong, and Jiani Hu. Generate adapt: Resolution adaption network for surveillance face recognition. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XV, volume 12360 of Lecture Notes in Computer Science, pages 741-758. Springer, 2020.
506
507
508
509
[55] Omkar M. Parkhi, Andrea Vedaldi, and Andrew Zisserman. Deep face recognition. In Xianghua Xie, Mark W. Jones, and Gary K. L. Tam, editors, Proceedings of the British Machine Vision Conference 2015, BMVC 2015, Swansea, UK, September 7-10, 2015, pages 41.1-41.12. BMVA Press, 2015.
510
511
512
513 Yi Sun, Yuheng Chen, Xiaogang Wang, and Xiaoou Tang. Deep learning face representation by joint ddentifiationn-veiiicattoo In Zoubin Ghahramani, Max Welling, Corinna Cortes, Neil D. I Lawrence, and Kilian Q Weinberger, editors, Advances in Neural Information Processing Systems 27: Annual Conference Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec. Canada, pages 1988-199 1996, 2014.
[56]
514
515
517
[57] Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pages 815-823. IEEE Computer Society, 2015.
519
520
521
522
(58) Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. Sphereface: Deep hypersphere embedding for face recognition. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017. pages 6738 6746. IEEE Computer Society, 2017.
523
524
525
526
[59]
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Learning deep convolutional network for image super-resolution. In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision ECCV 2014 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part IV, volume 8692 of Lecture Notes in Computer Science, pages 184-199. Springer, 2014.
529
530
531
[60]
Chao Dong, Chen Change Loy, and Xiaoou Tang. Accelerating the resouurnnolution convo- lutional neural network. In Bastian Leibe, Jiri Matas, Nieu Sebe, and Max Welling, editors, Computer Vision ECCV 2016 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II, volume 9906 of Lecture Notes in Computer Science pages 391-407. Springer, 2016
532
533
534
535
536
[61] Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. Accurate image super-resolution using very deep convolutional networks. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 1646-1654 IEEE Computer Society, 2016.
537
14

---

540
[62] Ying Tai, Jian Yang, and Xiaoming Liu. Image super-resolution via deep recursive residual network. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 2790-2798. IEEE Computer Society, 2017.
541
542
[63] Wei-Sheng Lai, Jia- Bin Huang, Narendra Ahuja, and Ming-Hsuan Yang. Deep laplacian pyramid networks for fast and accurate super-resolution. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 5835-5843.
544
545
5/6
IEEE Computer Society, 2017.
547
[64] 7. G. Liu and D. 7. Liu. Reappraising about image magnification methods based on wavelet transformation. Journal of Image and Graphics, 2003.
548
549
655
Wan-Chi Siu and Kwok-Wai Hung. Review of image interpolation and super-resolution. In Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA 2012, Hollywood, CA, USA, December 3-6, 2012, pages 1-10. IEEE, 2012.
552
[66]
Haohan Wang. Xindi Wu, Zeyi Huang, and Eric P. Xing. High-frequency component helps explain the generalization of convolutional neural networks. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 8681-8691. Computer Vision Foundation/ / IEEE, 2020.
553
555
Checklist
556
1. For all authors...
558
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contribu- tions and scope? [Yes]
559
560
(b) Did you descrihe the limitations of your work? [Yes] See Section 5 The dislocation between the underlying optical resolution of native face images and that of a certain branch may limit the power of the model, which may be improved by selecting the optimal processing branch for the input in combination with the image quality, rather than by image size alone. The optimal branch selection strategy is not fully investigated though we have provided an intuitive way to select the branch for inputs.
581
583
564
565
566
(c) Did you discuss any potential negative societal impacts of your work? [N/A] We study a general framework for multi- resolution face recognition. Our method is not for specific applications.
587
which does not directly involve societal issues.
589
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
570
2. If you are including theoretical results...
571
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Appendix A.1
572
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix A.
573
3. If you ran experiments...
574
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as URL)? [Yes]
575
576
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
577
[Yes]
578
(c) Did you report error bars (c.g., with respect to the random seed after running experiments multiple times)? [No] We follow the common practice in previous works, where they didn't 't report the error bars.
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See our implementation details in Section
582
583
4. If you are using existing assets (c.g., code, data, models) or curating/relcasngg new assets...
584
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
586
(c) Did you include any new assets either in the supplemental material or as URL? [No]
587
(d) Did you discuss whether and how consent was obtained from people whose data you're us-
ing/curating? [N/A]
589
(e) Did you discuss whether the data you are using/curating contains personally identifiable informa- tion or offensive content? [N/A]
15

---

5. If you used crowdsourcing or conducted research with human subjects...
592
(a) you include full text of instructions given to participants and screenshols, if applicable?
593
[N/A]
594
(b) Did you deseribe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
695
596
(C) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
587
16


Reviews:
[{"review_id": "lclDQG6XxEC", "reviewer": "Reviewer_NEKY", "summary": "This paper proposes a representation learning method, called Branch-to-Trunk network (BTNet), for multi-resolution face recognition. It consists of a trunk network (TNet) given in the form of a unified encoder, and multiple branch networks (BNets) that are used for resolution adaptation. A resolution-specific BNet is used on the input and the output are implanted as feature maps in the feature pyramid of TNet, at a layer with the same resolution. Using branch distillation and backward-compatible training, BTNet transfers high-resolution information to multiple branches, while keeping representation compatibility. Experiments show effective performance on face recognition benchmarks, both for multi-resolution identity matching and feature aggregation.", "questions": "Did the authors perform experiments on the branch selection process? Even preliminary tests would be useful.\n\nA figure reporting on overview of the proposed network it would have been useful.\n\nSee also weaknesses above. ", "limitations": "Limitations have been discussed by the authors.\n\nNo ethical issues have been reported. \n", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Code has been included in the submission supplementary files.\n- Experiments have been provided on several datasets. \n\nWeaknesses\nThe presentation of the results should be improved, Authors should better comment the different cases and propose explanation for specific situations where results are not intuitive in the tables.\n\nThe selection strategy for the optimal branch is not fully investigated. Author acknowledged this while exposing the limitations of their work, but I think this is something very relevant to the method that required more in depth understanding.\n\nPerformance outperform state-of-the-art on one dataset, while this does not occur on others. Authors should report motivations on why this happens. Which are the dataset characteristics that best explain this fact?\n\nThe paper writing needs some improvements. There is also some typos to check. See for example:\n- page 3, line 111: \"(3) This also effectively reduce\" --> reduces\n\n============= POST REBUTTAL ================\nThe rebuttal partially answered to my questions. I prefer to change my score to borderline accept. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Vk9dTtzXeHp", "reviewer": "Reviewer_RGyV", "summary": "In this paper, the authors introduce a Branch-to-Trunk Network (BTNet) for multi-resolution Face Recognition. \nParticularly, BTNet consists of two main components: BNet and TNet. While TNet aims at maintaining the main features flow (from high- to low-resolution), BNet is introduced for maps inputs to intermediate features with same resolution and maintain the feature compatibility when injecting them to the main flow of TNet.\nThe proposed BTNet is validated on QMUL-SurvFace and IJB-C protocols.", "questions": "1. The improvement achieved by BTNet is unclear: \nBTNet aims at training TNet for the main embedding flow, BNet aims at adopting the feature map extracted from the direct low-res images for making its features compatible to the feature flow of TNet. Moreover, only L_cur is adopted for the feature discriminative as [3]. \nHow can BTNet achieve improvements in comparison to prior works? How can it leave a large gap in comparison to phi_{mr} when same L_cur is adopted?\n\n2. Some baselines are missing:\n- In most experiments, phi_{hr}, phi_{mm}, and phi_{mr} are employed. However, in my opinion, these settings are quite “naive”, i.e. phi_{mr} is simply augmentation of multi-resolution images with similar weights for both HR and LR images. \nThey are acceptable for some basic baselines. The authors are recommended to have more baselines such as lower weight for LR images during training or enforce the features of LR images closer to the HR ones. \nThen, the baselines would be more complete.\n\n3. In Table 3, why is the accuracy of phi_{mm} higher than phi_{mr} in cross-resolution experiments? How about the compatibility of phi_{mm} in cross-resolution recognition? \n\n4. The comparisons in Table 3, 4 are lacking of prior works on IBJ-C.", "limitations": "Yes", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "STRENGTH\n- The paper is well-motivated.\n- The writing is easy to follow.\n- The idea of multi-branch network to overcome the limitation of the amount of information can be extracted from low-resolution input faces is interesting.\n- Experimental results show improvements in comparison to baselines and prior works.\n\n\nWEAKNESS\nGenerally, the novelty of the paper is limited as most of the building blocks (i.e. loss from CurricularFace [3] and common distillation loss) are introduced in previous works.\nAlthough the idea of a multi-branch network to overcome the robustness against low-resolution images, there are several concerns about this design. Please see the Question Section for further comments and questions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "3MU94jlfhe", "reviewer": "Reviewer_ra6W", "summary": "The paper proposes a Branch-to-Trunk network (BTNet) with multiple independent branch networks (BNets) and a shared trunk network (TNet) to extract feature representation for multi-resolution face recognition. The CurricularFace loss is refined as influence loss and a branch distillation loss is included for training to ensure the discriminative and compatible representation. The experiments conducted on both 1:1 and 1:N verification seems to validate the efficacy of the proposed method.", "questions": "See the above \"weakness\".", "limitations": "Yes.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n\n- The paper is well organized and easy to follow.\n\n- It is an interesting idea to decouple a face image as branches for discriminative representation learning and then couple them as the trunk for compatible representation learning. \n\n- The performance on the tasks of both cross-resolution and same-resolution identify matching looks very promising. \n\n\n### Weaknesses:\n\n- The authors claim the proposed BTNet is able to learn unified representation for face recognition. However, the experiment validation is only conducted on the face verification task. Will the learned unified representation can be beneficial to solve the general face recognition tasks?\n\n- The comparisons is inadequate to well demonstrate the superiority of the proposed BTNet.  All the baselines listed in Table 2 are at least two years ago.  The authors should compare the proposed BTNet with some recent face identification methods. \n\n- The necessary ablation studies are missing. In the current experimental results presentation, we only can see the overall performance gain. It is hard to guess which parts contribute more, and which parts contribute less. If the main performance gain is due to the introduction of CurricularFace loss as influence loss, then main technical contribution of TBNet may need to be doubted.\n\n- Lack the necessary qualitative results to show both successful and failure cases.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject