Sample and Computation Redistribution for Efficient Face Detection

1 INTRODUCTION . Face detection is a long-standing problem in computer vision with many applications , such as face alignment ( Bulat & Tzimiropoulos , 2017 ) , face reconstruction ( Feng et al. , 2018 ) , face attribute analysis ( Zhang et al. , 2018 ; Pan et al. , 2018 ) , and face recognition ( Schroff et al. , 2015 ; Deng et al. , 2019 ) . Following the pioneering work of ( Viola & Jones , 2004 ) , numerous face detection algorithms have been designed . Among them , the single-shot anchor-based approaches ( Najibi et al. , 2017 ; Zhang et al. , 2017b ; Tang et al. , 2018 ; Li et al. , 2019 ; Ming et al. , 2019 ; Deng et al. , 2020 ; Liu et al. , 2020 ; Zhu et al. , 2020 ) have recently demonstrated very promising performance . In particular , on the most challenging face detection dataset , WIDER FACE ( Yang et al. , 2016 ) , the average precision ( AP ) on its hard validation set has been boosted to 93.4 % by TinaFace ( Zhu et al. , 2020 ) . Even though TinaFace ( Zhu et al. , 2020 ) achieves impressive results on unconstrained face detection , it employs large-scale ( e.g . 1 , 650 pixels ) testing , which consumes huge amounts of computational resources . In addition , TinaFace design is based on a generic object detector ( i.e . RetinaNet ( Lin et al. , 2017b ) ) , directly taking the classification network as the backbone , tiling dense anchors on the multi-scale feature maps ( i.e . P2 to P7 of neck ) , and adopting heavy head designs . Without considering the prior of faces , the network design of TinaFace is thus redundant and sub-optimal . One approach of optimizing such networks ’ performance is computation redistribution . Since directly taking the backbone of the classification network for object detection is sub-optimal , the recent CR-NAS ( Liang et al. , 2020 ) reallocates the computation across different resolutions to obtain a more balanced Effective Receptive Field ( ERF ) , leading to higher detection performance . In BFbox ( Liu & Tang , 2020 ) , a face-appropriate search space is designed , based on the observation of scale distribution gap between general object detection and face detection . In ASFD ( Zhang et al. , 2020a ) , a differential architecture search is employed to discover optimized feature enhance modules for efficient multi-scale feature fusion and context enhancement . Even though ( Liu & Tang , 2020 ; Zhang et al. , 2020a ) have realized the limitation of directly applying general backbone , neck and head settings to face detection , CR-NAS ( Liang et al. , 2020 ) only focuses the optimization on backbone , BFbox ( Liu & Tang , 2020 ) neglects the optimization of head , and ASFD ( Zhang et al. , 2020a ) only explores the best design for neck . Another optimization approach , is the sample redistribution across different scales . Due to the extremely large scale variance of faces in real-world scenarios , different scale augmentation strategies are employed to introduce scale adaptation into the face detector . The most widely used scale augmentation approaches include random square crop ( Zhang et al. , 2017b ; Deng et al. , 2020 ; Zhu et al. , 2020 ) and data anchor sampling ( Tang et al. , 2018 ) . Nevertheless , the scale augmentation parameters in these methods are manually designed for all different network structures . Therefore , traditional multi-scale training in face detection is also tedious and sub-optimal . Since VGA resolution ( 640 × 480 ) is widely used for efficient face detection on numerous mobile phones and digital cameras , we focus on efficient face detection from low-resolution images in this paper . In Fig 1 ( a ) , we give the cumulative face scale distribution on the WIDER FACE validation dataset . Under the VGA resolution , most of the faces ( 78.93 % ) in WIDER FACE are smaller than 32×32 pixels . Under this specific scale distribution , both network structure and scale augmentation need to be optimized . In this work , we present a meticulously designed methodology of search space optimization , that addresses both the redistribution between the backbone , neck and head , and the sample redistribution between the most needed scales . As the structure of a face detector determines the distribution of computation and is the key in determining its accuracy and efficiency , we first discover principles of computation distribution under different flop regimes . Inspired by ( Radosavovic et al. , 2020 ) , we control the degrees of freedom and reduce the search space . More specifically , we randomly sample model architectures with different configurations on backbone ( stem and four stages ) , neck and head . Based on the statistics of these models , we compute the empirical bootstrap ( Efron & Tibshirani , 1994 ) and estimate the likely range in which the best models fall . To further decrease the complexity of the search space , we divide the computation ratio estimation for backbone and the whole detector into two steps . To handle extreme scale variations in face detection , we also design a search-able zoom-in and zoom-out space , specified by discrete scales and binary probabilities . In experiments , the proposed computation redistribution and sample redistribution yield significant and consistent improvement on various compute regimes , even surpassing a range of state-of-the-art face detectors by using much fewer flops as shown in Fig . 1 ( b ) . To sum up , this paper makes following contributions : • We have proposed a simplified search space , as well as a two-step search strategy for computation redistribution across different components ( backbone , neck and head ) of a face detector . The proposed computation redistribution method can easily boost detection performance through random search . • We have designed a search-able zoom-in and zoom-out space for face-specific scale augmentation , which automatically redistributes more training samples for shallow stages , enhancing the detection performance on small faces . • Extensive experiments conducted on WIDER FACE demonstrate the significantly improved accuracy and efficiency trade-off of the proposed SCRFD across a wide range of compute regimes . 2 RELATED WORK . Face Detection . To deal with extreme variations ( e.g . scale , pose , illumination and occlusion ) in face detection ( Yang et al. , 2016 ) , most of the recent single-shot face detectors focus on improving the anchor sampling/matching or feature enhancement . SSH ( Najibi et al. , 2017 ) builds detection modules on different feature maps with a rich receptive field . S3FD ( Zhang et al. , 2017b ) introduces an anchor compensation strategy by offsetting anchors for outer faces . PyramidBox ( Tang et al. , 2018 ) formulates a data-anchor-sampling strategy to increase the proportion of small faces in the training data . DSFD ( Li et al. , 2019 ) introduces small faces supervision signals on the backbone , which implicitly boosts the performance of pyramid features . Group sampling ( Ming et al. , 2019 ) emphasizes the importance of the ratio for matched and unmatched anchors . RetinaFace ( Deng et al. , 2020 ) employs deform-able context modules and additional landmark annotations to improve the performance of face detection . HAMBox ( Liu et al. , 2020 ) finds that many unmatched anchors in the training phase also have strong localization ability and proposes an online high-quality anchor mining strategy to assign high-quality anchors for outer faces . BFbox ( Liu & Tang , 2020 ) employs a single-path one-shot search method ( Guo et al. , 2019 ) to jointly optimize the backbone and neck for face detector . ASFD ( Zhang et al. , 2020a ) explores a differential architecture search to discover optimized feature enhance modules for efficient multi-scale feature fusion and context enhancement . All these methods are either designed by expert experience or partially optimized on backbone , neck and head . By contrast , we search for computation redistribution across different components ( backbone , neck and head ) of a face detector across a wide range of compute regimes . Neural Architecture Search . Given a fixed search space of possible networks , Neural Architecture Search ( NAS ) automatically finds a good model within the search space . DetNAS ( Chen et al. , 2019b ) adopts the evolution algorithm for the backbone search to boost object detection on COCO ( Lin et al. , 2014 ) . By contrast , CR-NAS ( Liang et al. , 2020 ) reallocates the computation across different stages within the backbone to improve object detection . NAS-FPN ( Ghiasi et al. , 2019 ) uses reinforcement learning to search the proper FPN for general object detection . As there is an obvious distribution gap between COCO ( Lin et al. , 2014 ) and WIDER FACE ( Yang et al. , 2016 ) , the experience in the above methods is not directly applicable for face detection but gives us an inspiration that the backbone , neck and head can be optimized to enhance the performance of face detection . Inspired by RegNet ( Radosavovic et al. , 2020 ) , we optimize the computation distribution on backbone , neck and head based on the statistics from a group of random sampled models . We successfully reduce the search space and find the stable computation distribution under a particular complex regime , which significantly improves the model ’ s performance . 3 METHODOLOGY . To efficiently and accurately detect small faces from low-resolution images ( e.g . VGA 640 × 480 ) , we propose two methodologies that , when combined , outperform the state-of-the-art . In Sec . 3.1 , we explore the computation redistribution across different stages of backbone , as well as different components ( i.e . backbone , neck and head ) of the whole detector , given a pre-defined computation budget . Then , in Sec . 3.2 , we investigate the redistribution of positive training samples across different scales of feature maps by searching optimized scale augmentations . 3.1 COMPUTATION REDISTRIBUTION . As illustrated in Fig . 2 , we apply our search method on a network consisting of ( 1 ) RetinaNet ( Lin et al. , 2017a ) , with ResNet ( He et al. , 2016 ) as the backbone , ( 2 ) Path Aggregation Feature Pyramid Network ( PAFPN ) ( Liu et al. , 2018 ) as the neck , and ( 3 ) stacked 3 × 3 convolutional layers for the head . Despite the generally simple structure , the total number of possible network configurations of the search space becomes unwieldy . Therefore , we attempt to simplify the tremendous search space and arrive at a low-dimensional design space , consisting of simple and effective networks . 3.1.1 SEARCH SPACE DESIGN . Inspired by RegNet ( Radosavovic et al. , 2020 ) , we explore the structures of face detectors , assuming fixed standard network blocks ( i.e. , basic residual or bottleneck blocks with a fixed bottleneck ratio of 4 ) . In our case , the structure of a face detector includes : • the backbone stem , three 3× 3 convolutional layers with w1 output channels ( He et al. , 2019a ) . • the backbone body , four stages ( i.e . C2 , C3 , C4 and C5 ) operating at progressively reduced resolution , with each stage consisting of a sequence of identical blocks . For each stage i , the degrees of freedom include the number of blocks di ( i.e . network depth ) and the block width wi ( i.e . number of channels ) . • the neck , a multi-scale feature aggregation module by a top-down path and a bottom-up path with n channels ( Liu et al. , 2018 ) . • the head , with hi channels of m blocks to predict face scores and regress face boxes . The search space can be initially designed as follows . As the channel number of the stem is equal to the block width of the first residual block in C2 , the degree of freedom of the stem w1 can be merged into w2 . In addition , we employ a shared head design for three-scale of feature maps and fix the channel number for all 3 × 3 convolutional layers within the heads . Therefore , we reduce the degrees of freedom to three within our neck and head design : ( 1 ) output channel number n for neck , ( 2 ) output channel number h for head , and ( 3 ) the number of 3 × 3 convolutional layers m. We perform uniform sampling of n ≤ 256 , h ≤ 256 , and m ≤ 6 ( both n and h are divisible by 8 ) . The backbone search space has 8 degrees of freedom as there are 4 stages and each stage i has 2 parameters : the number of blocks di and block width wi . Following RegNet ( Radosavovic et al. , 2020 ) , we perform uniform sampling of di ≤ 24 and wi ≤ 512 ( wi is divisible by 8 ) . As state-ofthe-art backbones have increasing widths ( Radosavovic et al. , 2020 ) , we also constrain the search space , according to the principle of wi+1 ≥ wi .

Summary: This paper presents a face detection method that aims to deal with two challenges in unconstrained face detection: high computation cost and detecting small faces. Specifically, the authors adopted a network structure search method along with a two-step searching strategy for computation redistribution between the backbone, neck and head of the network. Sample redistribution between the scales is achieved using a searchable zoom-in and zoom-out space for face scale augmentation. Experiments were conducted on a benchmark dataset: WIDER FACE.